Schachtman Law » Reference Manual on Scientific Evidence

TORTINI

For your delectation and delight, desultory dicta on the law of delicts.

The Reference Manual’s Chapter on Expert Witness Testimony Admissibility – Part One

February 23rd, 2026

With the retraction of the climate science chapter, The Reference Manual on Scientific Evidence is now one chapter shorter, at least in the Federal Judicial Center’s version. At the time of this writing, for curious souls, the National Academies version is still sporting the climate advocacy chapter. Even without the climate chapter, the Manual is over 1,000 pages, and more than a casual weekend read. Many judges, finding this tome on their desks, will read individual subject matter chapters pro re nata. The first chapter in the Manual, however, is about the law, not science, and might be the starting place for the ordinary work-a-day judge. As in past editions of the Manual, the new edition has a chapter on the The Admissibility of Expert Testimony. In the first, second, and third editions, this chapter was written by Professor Margaret Berger. In the fourth edition, the chapter on the law was written by law professors Liesa Richter and Daniel Capra. To understand and evaluate the most recent iteration, the reader should have some sense of what has gone before.

Previous Chapters on Admissibility of Expert Witness Testimony

Professor Berger’s past chapters had been idiosyncratic productions.[1] Berger was an evidence law scholar, who wrote often about expert witness admissibility issues.[2] She was also known for her antic proposals, such as calling for abandoning the element of causation in products liability cases.[3] As an outspoken ideological opponent of expert witness gatekeeping, Berger was a strange choice to write the law chapter of the Manual.[4] Berger’s chapters in the first through the third editions made her opposition to gatekeeping obvious, and this hostility may have been responsible for some of the judicial resistance to applying the clear language of Rule 702, even after its 2000 revision.

Berger was not only a law professor; she was at the center of ideological and financially conflicted groups that worked to undermine the application of Rule 702 in health effects cases. One of the key players in this concerted action was David Michaels. Currently, Michaels teaches epidemiology at the George Washington University Milken Institute School of Public Health. He is a card-carrying member of the Collegium Ramazzini, an organization that has participated in efforts to corrupt state and federal judges by funding ex parte conferences with lawsuit industry expert witnesses.[5] Michaels is the author of two books, both highly anti-manufacturing industry, and biased in favor of the lawsuit industry.[6] Both books are provocatively titled anti-industry diatribes, which have little scholarly value, but are used regularly by plaintiffs counsel solely to smear corporate defendants and defense expert witnesses. Most clear-eyed trial judges have quashed these efforts on various grounds, including Rule 703, because the books are not the sort of material upon which scientists would reasonably rely.[7]

In 2002, David Michaels created an anti-Daubert advocacy organization, the Project on Scientific Knowledge and Public Policy (SKAPP), from money siphoned from the plaintiffs’ common-benefit fund in MDL 926 (silicone gel breast implant litigation).[8] Michaels lavished some of the misdirected money to prepare and publish an anti-Daubert pamphlet for SKAPP, in 2003.[9] In this anti-Daubert publications, and many others sponsored by SKAPP, Michaels and the SKAPP grantees typically acknowledged the source of SKAPP funding obliquely to hide that it was nothing more than plaintiffs counsels’ walking around money:

“I am also grateful for the support SKAPP has received from the Common Benefit Trust, a fund established pursuant to a court order in the Silicone Gel Breast Implant Liability litigation.”[10]

Many credulous lawyers, judges, and legal scholars were duped into believing that SKAPP, SKAPP publications, and SKAPP-sponsored publications were supported by the Federal Judicial Center.

Michaels directed a good amount of SKAPP’s anti-Daubert funding to support Professor Berger’s efforts in organizing a series of symposia on science and the law. Several of Berger’s SKAPP conferences were held in Coronado, California, and featured a predominance of scientists who work for the lawsuit industry and are affiliated with advocacy organizations, such as the Collegium Ramazzini. The papers from one of the Coronado Conferences were published in a special issue of the American Journal of Public Health, the official journal of the American Public Health Association,[11] which has issued position papers highly critical of Rule 702 gatekeeping.[12]

The spider web of connections between SKAPP, the Collegium Ramazzini, the American Public Health Association, the Tellus Institute, the lawsuit industry, Professor Berger, and others hostile to Rule 702 is a testament to the concerted action to undermine the Supreme Court’s decisions in the area, and the codification of those decisions in Rule 702. That Professor Berger was within this web of connections, and was writing the chapter on the admissibility of expert witness opinion testimony, in the first three editions of the Reference Manual, explains but does not justify many of the opinions contained within those chapters.

Professor David Bernstein, who has written extensively on expert witness issues, restated the situation thus:

“In 2003, the toxic tort plaintiffs’ bar used money from a fund established as part of the silicone breast implant litigation settlement to sponsor four conference in Coronado, California, that resulted in a slew of policy papers excoriating the Daubert gatekeeping requirement.”[13]

The active measures of these groups and Professor Berger explain the straight line between Berger’s symposia and the First Circuit’s decision in Milward v. Acuity Specialty Products Group, Inc.[14] Carl Cranor was one of the speakers at the Coronado Conferences, and along with Martyn Smith, another member of the Collegium Ramazzini, founded a Proposition 65 bounty-hunting organization, Council for Education on Research on Toxics (CERT). Cranor has long advocated for a loosey-goosey “weight of the evidence” approach that had been rejected by the Supreme Court in Joiner.[15] Cranor, along with Smith, unsurprisingly turned up as expert witnesses for plaintiff in Milward, in which case they reprised their weight-of-the evidence approach opinions. When Milward appealed the exclusion of Cranor and Smith, CERT filed an amicus brief, without disclosing that Cranor and Smith were founders of the organization, and that CERT funded Smith’s research through donations to his university, from CERT’s shake-down operations under Prop 65. The First Circuit’s 2011 decision in Milward resulted from a fraud on the court.

Professor Berger died in November 2010, but when the third edition of the Manual was released in 2011, it contained Berger’s chapter on the law of expert witnesses, with a citation to the Milward case, decided after her death.[16] An editorial note from an unnamed editor to her posthumous chapter suggested that

“[w]hile revising this chapter Professor Berger became ill and, tragically, passed away. We have published her last revision, with a few edits to respond to suggestions by reviewers.”

Given that Berger was an ideological opponent of expert witness gatekeeping, there can be little doubt that she would have endorsed the favorable references to Milward made after her passing, but adding them can hardly be considered non-substantive edits. Curious readers might wonder who was the editor who took such liberties of adding the chapter citations to Milward. Curious readers do not have to wonder, however, what would have happened if the incestuous relationships among Berger, SKAPP, the plaintiffs’ bar, and others had been replicated by similar efforts of manufacturing industry to influence the interpretation and application of the law. In 2008, the Supreme Court decided an important case involving constitutional aspects of punitive damages. The Court went out of its way to decline to rely upon empirical research that showed the unpredictability of punitive damage awards because it was funded in part by Exxon:

“The Court is aware of a body of literature running parallel to anecdotal reports, examining the predictability of punitive awards by conducting numerous ‘mock juries’, where different ‘jurors’ are confronted with the same hypothetical case. See, e.g., C. Sunstein, R. Hastie, J. Payne, D. Schkade, & W. Viscusi, Punitive Damages: How Juries Decide (2002); Schkade, Sunstein, & Kahneman, Deliberating About Dollars: The Severity Shift, 100 Colum. L.Rev. 1139 (2000); Hastie, Schkade, & Payne, Juror Judgments in Civil Cases: Effects of Plaintiff’s Requests and Plaintiff’s Identity on Punitive Damage Awards, 23 Law & Hum. Behav. 445 (1999); Sunstein, Kahneman, & Schkade, Assessing Punitive Damages (with Notes on Cognition and Valuation in Law), 107 Yale L.J. 2071 (1998). Because this research was funded in part by Exxon, we decline to rely on it.”[17]

Unlike the situation with SKAPP, David Michaels, the plaintiffs’ bar, and Professor Berger, the studies sponsored in part by Exxon had disclosed their funding clearly. Those studies involved outstanding scientists whose integrity were unquestionable, and for its trouble, Exxon was rewarded with gratuitous shaming from Justice Souter. The anti-Daubert papers sponsored by the plaintiffs’ bar through SKAPP, and Professor Berger’s ideological conflicts of interest have received a free pass. This disparate treatment between conflicts of interest within manufacturing industry and those within the lawsuit industry and its advocacy group allies is a serious social, political, and legal problem. It was a problem on full display in the now-retracted climate science chapter in the Manual. In evaluating the new fourth edition’s chapter on the law of expert witness admissibility (and other chapters), we should be asking whether there are signs of undue political influence.

[1] See Schachtman, The Late Professor Berger’s Introduction to the Reference Manual on Scientific Evidence, TORTINI (Oct. 23, 2011).

[2] See generally Edward K. Cheng, Introduction: Festschrift in Honor of Margaret A. Berger, 75 BROOKLYN L. REV. 1057 (2010).

[3] Margaret A. Berger, Eliminating General Causation: Notes towards a New Theory of Justice and Toxic Torts, 97 COLUM. L. REV. 2117 (1997).

[4] See, e.g., Margaret A. Berger & Aaron D. Twerski, “Uncertainty and Informed Choice: Unmasking Daubert,” 104 MICH. L. REV. 257 (2005).

[5] In re School Asbestos Litig., 977 F.2d 764 (3d Cir. 1992). See Cathleen M. Devlin, Disqualification of Federal Judges – Third Circuit Orders District Judge James McGirr Kelly to Disqualify Himself So As To Preserve ‘The Appearance of Justice’ Under 28 U.S.C. § 455 – In re School Asbestos Litigation (1992), 38 VILL. L. REV. 1219 (1993); Bruce A. Green, May Judges Attend Privately Funded Educational Programs? Should Judicial Education Be Privatized?: Questions of Judicial Ethics and Policy, 29 FORDHAM URB. L. J. 941, 996-98 (2002).

[6] David Michael, DOUBT IS THEIR PRODUCT: HOW INDUSTRY’S WAR ON SCIENCE THREATENS YOUR HEALTH (2008); David Michaels, THE TRIUMPH OF DOUBT (2020).

[7] See In re DePuy Orthopaedics, Inc. Pinnacle Hip Implant Prods. Liab. Litig., 888 F.3d 753, 787 n.71 (5th Cir. 2018) (advising the district court to weigh carefully whether Doubt is Their Product has any legal relevance); King v. DePuy Orthopaedics, Inc., 2024 WL 6953089, at *2 (D. Ariz. July 9, 2024) (finding Michaels’ books to be legally irrelevant); Sarjeant v. Foster Wheeler LLC, 2024 WL 4658407, at *1 (N.D. Cal.Oct. 24, 2024) (ruling that Doubt Is Their Product is legally irrelevant hearsay, and not the type of material upon which an expert witness would rely to form scientific opinion). See also Evans v. Biomet, Inc., 2022 WL 3648250, at *4 (D. Alaska Feb. 1, 2022) (quashing plaintiff’s subpoena to defendant’s expert for material in connection with Doubt Is Their Product).

[8] See Ralph Klier v. Elf Atochem North America Inc., 2011 U.S. App. LEXIS 19650 (5th Cir. 2011) (holding that district court abused its discretion in distributing residual funds from class action over arsenic exposure to charities; directing that residual funds be distributed to class members with manifest personal injuries). A “common benefit” fund is commonplace in multi-district litigation of mass torts. In such cases, federal courts may require the defendant to “hold back” a certain percentage of settlement proceeds, to pay into a fund, which is available to those plaintiffs’ counsel who did “common benefit work,” work for the benefit of all claimants. Plaintiffs’ counsel who worked for the common benefit of all claimants may petition the MDL court for compensation or reimbursement for their work or expenses. See, e.g., William Rubenstein, On What a ‘Common Benefit Fee’ Is, Is Not, and Should Be, CLASS ACTION ATT’Y FEE DIG. 87, 89 (Mar. 2009). In the silicone gel breast implant litigation (MDL 926), plaintiffs’ counsel on the MDL Steering Committee undertook common benefit work in the form of developing expert witnesses for trial, and funding scientific studies. By MDL Orders 13, and 13A, the Court set hold-back amounts of 5 or 6%, and later reduced the amount to 4%. Id. at 94.

[9] Eula Bingham, Leslie Boden, Richard Clapp, Polly Hoppin, Sheldon Krimsky, David Michaels, David Ozonoff & Anthony Robbins, Daubert: The Most Influential Supreme Court Ruling You’ve Never Heard Of (June 2003). The authors described the publication as a publication of SKAPP, coordinated by the Tellus Institute, and funded by The Bauman Foundation, a private foundation that supports “progressive social change advocacy.” Boden, Hoppin, Michaels, and Ozonoff are fellows of the Collegium Ramazzini.

[10] David Michael, DOUBT IS THEIR PRODUCT: HOW INDUSTRY’S WAR ON SCIENCE THREATENS YOUR HEALTH 267 (2008). See Nathan Schachtman, “SKAPP A LOT,” TORTINI (April 30, 2010); “Manufacturing Certainty” TORTINI (Oct. 25, 2011); “David Michaels’ Public Relations Problem” TORTINI (Dec. 2, 2011); “Conflicted Public Interest Groups” TORTINI (Nov. 3, 2013).

[11] 95 AM. J. PUB. HEALTH S1 (2005).

[12] See, e.g., Am. Pub. Health Assn, Threats to Public Health Science, Policy Statement 2004-11 (Nov. 9, 2004), available at https://www.apha.org/policy-and-advocacy/public-health-policy-briefs/policy-database/2014/07/02/08/52/threats-to-public-health-science

[13] David E. Bernstein & Eric G. Lasker, Defending Daubert: It’s Time to Amend Federal Rule of Evidence, 702, 57 WM. & MARY L. REV. 1, 39 (2015), available at https://scholarship.law.wm.edu/wmlr/vol57/iss1/2. See David Michaels & Neil Vidmar, Foreword, 72 LAW & CONTEMP. PROBS. i, ii (2009) (“SKAPP has convened four Coronado Conferences.”).

[14] Milward v. Acuity Specialty Products Group, Inc., 639 F.3d 11 (1st Cir. 2011), cert. denied sub nom., U.S. Steel Corp. v. Milward, 132 S. Ct. 1002 (2012).

[15] General Electric Co. v. Joiner, 522 U.S. 136, 136-37 (1997).

[16] Margaret A. Berger, The Admissibility of Expert Testimony, in National Academies of Sciences, Engineering and Medicine & Federal Judicial Center, REFERENCE MANUAL ON SCIENTIFIC EVIDENCE 11, 20 n.51, 23-24 n.61 (3rd 2011).

[17] Exxon Shipping Co. v. Baker, 554 U.S. 471, 128 S. Ct. 2605, 2626 n.17 (2008).

Posted in Conflicts of Interest, Expert Witnesses, Reference Manual on Scientific Evidence, Rule 702, Uncategorized | No Comments »

The FJC Retracts Climate Science – Postscriptum

February 11th, 2026

The version of the Reference Manual on the NASEM website still has the climate science chapter.

The FJC website has a version without the climate science chapter. There is a note that the chapter was removed on February 6, 2026.

Perhaps the two organizations should talk?

The FJC retraction has been covered by many media outlets, but the failure of NASEM to act has not been reported, as far as I can see.[1]

[1] See, e.g., Nate Raymond, US judiciary scraps climate chapter from scientific evidence manual, REUTERS (Feb. 9, 2026)

Posted in Reference Manual on Scientific Evidence | 1 Comment »

The FJC Retracts Climate Science Chapter in New Reference Manual

February 10th, 2026

When the new, fourth, edition of the Reference Manual on Scientific Evidence was released late last year,[1] I remarked that there were some new chapters,[2] including one on climate change. I found the addition of a chapter on climate change curious largely because I was unfamiliar with the science or the need to address the area for federal judges, and because I thought there were other more pressing topics, such as genetic causation, from which judges could benefit, but which were not included.

I confess that I did not read the new chapter on climate change,[3] which is not a subject that comes up in my practice or in my writing. Writers at the National Review, however, did read the chapter on climate, and found it objectionable. Writing on January 17^th of this year, Michael Fragoso observed that the chapter on climate science was an advocacy piece that would resolve climate change litigation in favor of plaintiffs.[4]

If Fragoso’s charge is correct, the implications are extremely serious. Judges have an ethical obligation not to go beyond the adversary process to educate themselves about the factual issues before them in pending litigation. In the past, judges who have done so have found themselves on the wrong end of a petition for a writ of mandamus, and have been disqualified and removed from cases.[5] The Federal Judicial Center (FJC), which is the research and educational division of the federal courts, has tried to create a safe space for teaching judges about technical subjects that arise in litigation in a way that is balanced and removed from partisan advocacy. The last edition, the third, and the current edition, the fourth, of the Manual have been the joint product of the both the FJC and the National Academies of Science, Engineering and Medicine (NASEM), in the hope of producing disinterested tutorials on key areas of science that are important to judges in their adjudication of civil and criminal cases, as well as their performance of judicial review of regulation and agency action.

Following up on the National Review article, on January 29, 2026, the Attorneys General of 24 states[6] wrote a letter to Judge Robin Rosenberg, the director of the Federal Judicial Center. The letter identified the advocacy perspective of the climate chapter and its authors, who wrote what the Attorneys General described as an amicus brief that placed a thumb on the scales of justice, with respect to issues currently pending at all levels of the federal courts. The Attorneys General requested the immediate withdrawal of the offending chapter.

Judge Rosenberg is a savvy judge of scientific horse flesh. She presided over the Zantac multi-district litigation (MDL No. 2924), in which she excluded plaintiffs’ expert witnesses in a detailed, analytically careful opinion of over 300 pages.[7] On February 6, a week after the request to withdraw the climate chapter was made, Judge Rosenberg, wrote to West Virginia Attorney General John McCuskey, to report that the chapter had been omitted.[8] Given the prompt response from Judge Rosenberg, the decision was likely not a difficult one. A decision not to include this chapter, as written, in the first place, would have been an even easier one.

Retractions of publications of the NASEM, which includes what was formerly the Institute of Medicine, are rarer than hens teeth. This one received coverage and some intense harrumphing.[9] The retraction of the climate science chapter comes on the heels of a high-profile retraction, in December 2025, of an article in the prestigious journal Nature,[10] which argued that the costs of climate change would reach $38 trillion a year by 2049.[11]

The climate science chapter appears to be the outcome of what the late Daniel Kahneman called poor decision hygiene. The chapter in question had two authors, and both were from the same institution, published together, and shared the same advocacy perspectives on climate change. Hardly a team of rivals. The editors of the Manual certainly could have done better in selecting these authors and in editing the work product.

Jessica Wentz is a Non-Resident Senior Fellow at the Sabin Center for Climate Change Law, at the Columbia Law School. The Sabin Center website describes itself as “develop[ing] legal techniques to combat the climate crisis and advance climate justice, and train the next generation of leaders in the field.” The language of “combat” and “crisis” certainly suggests a hardened, adversarial stance. Wentz’s writings reveal her advocacy and adversarial positions.[12]

Radley Horton is a Professor at Columbia University’s Climate School. He describes his research as focusing on climate extremes, and related topics. Horton’s curriculum vitae, social media, social media, testimony,[13] and professional work certainly mark him as an advocate for “attribution science” in litigation to address climate crises. Horton and Wentz previously published a law review article that seems to be a brief for plaintiffs’ positions in climate litigation.[14] One of the key issues in climate litigation is whether litigation is an appropriate avenue for addressing climate issues, and Horton and Wentz have both clearly committed to endorsing litigation strategies, and the plaintiffs’ positions to boot.

This kerfuffle at FJC and NASEM has a larger meaning. There is a glib assumption afoot that the only conflicts of interest that matter are ones that are attributed to industrial stakeholders and their scientific supporters. This naïve view was attacked and debunked back in 1980, by Sir Richard Peto, writing in the pages of Nature. Sir Richard noted that whereas industry may downplay risks, “environmentalists usually exaggerate the likely hazards and are largely indifferent to the costs of control.” Positional conflicts can be, and often are, more powerful than the ones created by profit.[15]

[1] National Academies of Sciences, Engineering, and Medicine & Federal Judicial Center, REFERENCE MANUAL ON SCIENTIFIC EVIDENCE (4^th ed. 2025) (cited as RMSE 4^th ed.).

[2] Nathan Schachtman, A New Year, A New Reference Manual, in TORTINI (Jan. 5, 2026).

[3] Jessica Wentz & Radley Horton, Reference Guide on Climate Science, RMSE 4^th ed.

[4] Michael A. Fragoso, Bias and the Federal Judicial Center’s ‘Climate Science’, NAT’L REV. (Jan. 17, 2026). Fragoso also took umbrage to the use of the silly phrase “pregnant people” elsewhere in the Manual. RMSE 4^th ed. at 84.

[5] In re School Asbestos Litigation, 977 F.2d 764 (3d Cir. 1992). See Cathleen M. Devlin, Disqualification of Federal Judges – Third Circuit Orders District Judge James McGirr Kelly to Disqualify Himself So As To Preserve ‘The Appearance of Justice’ Under 28 U.S.C. § 455 – In re School Asbestos Litigation (1992), 38 VILL. L. REV. 1219 (1993);

[6] Alabama, Alaska, Arkansas, Florida, Georgia, Idaho, Indiana, Iowa, Missouri, Montana, Nebraska, New Hampshire, North Dakato, Ohio, Oklahoma, Pennsylvania, South Carolina, South Dakato, Tennessee, Texas, Utah, West Virginia, West Virginia, and Wyoming.

[7] In re Zantac (Ranidine) Prods. Liab. Litig., 644 F. Supp. 3d 1075 (S.D. Fla. 2022).

[8] Hon. Robin Rosenberg, Letter in Response to Attorneys General (Feb. 6, 2026).

[9] Editorial Board, A Failed Climate Coup in the Courts, WALL ST. J. (Feb. 9, 2026); Charles Creitz, Judicial research center cuts climate section from judges’ manual, FOX NEWS (Feb. 9, 2026); Suzanne Monyak, Judiciary Cuts Climate Part of Science Manual after Backlash, BLOOMBERG LAW (Feb. 9, 2026).

[10] Maximilian Kotz, Anders Levermann & Leonie Wenz, The economic commitment of climate change, 628 NATURE 551 (2024) (retracted on Dec. 3, 2025).

[11] Authors retract Nature paper projecting high costs of climate change, RETRACTION WATCH (Dec. 3, 2025).

[12] Michael Burger, Jessica Wentz & Daniel Metzger, Climate science in rights-based advocacy contexts (June 28, 2020).

[13] Written Testimony of Radley Horton, Lamont Associate Research Professor, Columbia University, before the Committee on Science, Space, and Technology Subcommittee on Environment Sea Change: Impacts of Climate Change on Our Oceans and Coasts (Feb. 27, 2019).

[14] Michael Burger, Radley M. Horton & Jessica Wentz, The Law and Science of Climate Change Attribution, 45 COLUMBIA J. ENVT’L J. L. 57 (2020).

[15] Richard Peto, Distorting the epidemiology of cancer: the need for a more balanced overview, 284 NATURE 297, 297 (1980).

Posted in Cognitive Biases, Conflicts of Interest, Reference Manual on Scientific Evidence | No Comments »

The 4th Reference Manual’s Treatment of Genetic Causes of Disease

January 23rd, 2026

After checking to see whether the new Reference Manual on Scientific Evidence[1] attended to some long overdue corrections, I turned my attention to the substance of the chapter on epidemiology. A cursory comparison between the third[2] and fourth[3] editions of the epidemiology chapter in the Reference Manual a lot of carry over from the third edition, some change in authorship, and at least one interesting change.

The two lawyer authors, Steve Gold and Michael Green, remain, but the authors with reasonable pretense to subject-matter expertise have changed. Gold and Green are both law professors with a long history of commenting on American tort and evidence law. Both are aligned with the lawsuit industry. Previous epidemiology authors, Daryl Michal Freedman and Leon Gordis are now gone from the chapter. Leon Gordis, who had been a chairman of the department of epidemiology, in the Bloomberg School of Public Health, Johns Hopkins University, died in September 2015, after the third edition was published. Daryl Michal Freedman, who been the other subject-matter expert on the third edition’s chapter on epidemiology, has been an epidemiologist with the Biostatistics Branch of the National Cancer Institute, for many years. It is not clear why he left the project.

Replacing Gordis and Freedman are Jonathan Chevrier and Brenda Eskenazi. Chevrier is an associate professor on the faculty of medicine, in the department of epidemiology, in McGill University. The focus of his work is on “common environmental contaminants,” and the role in the development and health of children. Brenda Eskenazi is professor emerita, in the University of California Berkeley School of Public Health, where she is the Director of the Center for Environmental Research and Children’s Health. Eskenazi is a member of a dodgy group known as the Collegium Ramazzini, which was responsible for staging an ex parte presentation of plaintiffs’ expert witnesses to judges presiding in asbestos litigation.[4] Eskenazi was not, however, a member of the Collegium at the time the group conspired with the late Irving Selikoff to pervert the course of justice in American asbestos litigation.

The second significant change is substantive; the fourth edition has added a new subsection to the epidemiology chapter. Comparing the texts of the third and fourth editions of this chapter reveals a new subheading in the new edition:[5]

Genetic and Molecular Epidemiologic Studies

Alas, there is not as much substance to the new subsection, which is less than four pages. Lawyers in the trenches might well have hoped for more substantive treatment of genetic epidemiology, and genetic causation. The chapter’s authors explain their abbreviated treatment with the comment:

“Although commentators have long forecast that the output of genetic and molecular epidemiology would revolutionize causal proof, as of this writing few judicial opinions have addressed these types of studies, and it is far from clear that a revolution is in the offing.”[6]

The chapter authors are correct that some authors in the past proffered unrealistic predictions of how genetics would supplant correlational studies. Nonetheless, this area has not been as quiescent as the authors’ parsimonious treatment would suggest.

On the question of how prevalent are genetic causation issues, whether raised by plaintiffs or defendants, the chapter might have benefitted from the contributions of a practicing lawyer. Genetic issues come up with some frequency in the litigation of cases involving mesothelioma. The days of plaintiffs who had 30 years of amphibole asbestos exposure in the workplace are largely over. Today’s cases involve little to no exposure, and it stands to reason that the origins of the recently diagnosed cases are different from those diagnosed in the 1970s and 1980s.[7] Genetic cause of mesothelioma is a salient current issue that is passed over in this new Reference Manual.

The authors acknowledge a single birth defects case in which genetic causation was litigated,[8] which was already old news when the last edition of the Manual was published. There are now many more reported cases that cry out for discussion in this under-covered area of the Manual.[9] There are also many cases not reported that have turned on genetic issues. For instance, in some cancer and birth defect cases, the existence of a highly penetrant genetic mutation that could explain the occurrence of a disease completely raises a serious question whether the plaintiff who fails to test for the mutation can possibly have carried his burden of proof.[10] And then there are myriad cases in which the parties have engaged in motion practice, sometimes extended, over access to genetic testing materials.

Genetic issues have arisen in the litigation of high-profile general causation disputes. For instance, the failure to control for genetic effects in epidemiologic studies was a significant issue in the acetaminophen-autism litigation, with both sides presenting geneticists to explain whether the relevant studies were undermined by failure to control for genetic effects.[11]

In the Manual’s epidemiology chapter’s new section on genetics, the authors describe some basic terms and explain that genetic epidemiology may provide evidence for, or against, claims of health effects. The authors’ views come through most clearly in the following short passage:

“Alternatively, genetic epidemiology may reveal associations between genetic variations and a plaintiff’s disease, raising the issue of whether or not a genetic variation may be a competing cause of the disease. This requires assessment of whether the gene–disease association is causal in a general sense, whether it acts independently of the exposure, and whether it is a competing cause in the plaintiff’s specific instance. The extreme, though not typical, example would be a health outcome or disease entirely determined by genetics, ⁵⁵ as is the case with sickle cell anemia.⁵⁶”[12]

The authors never explain or defend their claim that cases involving diseases caused entirely by genetics are “extreme” and “not typical.” At several points, the authors emphasize that gene-environment interactions are the more prevalent determinants of diseases.[13] If we were to catalog the currently known genetic determinants of diseases, the authors may be correct on a percentage basis, but the issue in any given case is whether the disease or harm claimed by the plaintiff is one of the “extreme” cases of complete genetic causation, or an instance of genetic susceptibility. The authors’ generalization, even if it were correct, would not be very helpful or informative for any specific case.

Perhaps even more important for lawyers, there is a substantive issue on which the new chapter manages to provide confusing guidance. The epidemiology chapter appears to create a false dichotomy between rare, highly penetrant genetic mutations that are uncommon causes of certain diseases, and the more prevalent genetic mutations and polymorphisms that leave persons more susceptible to the deleterious effects of exogenous exposures to toxic chemicals.[14] There is, however, another scenario omitted in the chapter’s discussion of genetic causation. Genetic mutations and polymorphisms may leave persons susceptible to normal, endogenous chemicals, stochastic cellular events, and biological processes that result in diseases such as cancers. In other words, the knee-jerk reflex to invoke exogenous, external toxic chemical exposures promotes a false dichotomy and obscures the obvious implication that susceptibility mutations and polymorphisms may lead to cancer without environmental exposures to harmful chemicals.[15]

The number of endogenous events leading to DNA alterations is enormous, and requires us to rethink the mantra that attributes chronic diseases to gene-environment interaction. At the very least, we need to stop thinking of “environment” as chemical exposures from without ourselves. The epidemiology chapter authors, like many writers, point to external chemical exposures as the culprits in gene-environment reactions, but they ignore the normal, endogenous events that lead to DNA damage, for which genetic susceptibility may be relevant. Mutations that result in increased susceptibility to cancer may affect DNA alterations from both endogenous and metabolic factors as well as from exposures to external chemicals.

Ignorance is never a good thing, and the chapter does the bar and bench a disservice in not adequately exploring genetic susceptibility in view of both exogenous and endogenous exposures that may be responsible for chronic diseases, such as cancers.

[1] National Academies of Sciences, Engineering, and Medicine & Federal Judicial Center, REFERENCE MANUAL ON SCIENTIFIC EVIDENCE (4^th ed. 2025) (cited as RMSE 4^th ed.)

[2] Michael D. Green, D. Michal Freedman & Leon Gordis, Reference Guide on Epidemiology, 549, in RMSE 3^rd ed.

[3] Steve C. Gold, Michael D. Green, Jonathan Chevrier, & Brenda Eskenazi, Reference Guide on Epidemiology, in RMSE 4^th ed.

[4] See In re School Asbestos Litigation, 977 F.2d 764 (3d Cir. 1992). See also Cathleen M. Devlin, Disqualification of Federal Judges – Third Circuit Orders District Judge James McGirr Kelly to Disqualify Himself So As To Preserve ‘The Appearance of Justice’ Under 28 U.S.C. § 455 – In re School Asbestos Litigation (1992), 38 VILL. L. REV. 1219 (1993); Bruce A. Green, May Judges Attend Privately Funded Educational Programs? Should Judicial Education Be Privatized?: Questions of Judicial Ethics and Policy, 29 FORDHAM URB. L. J. 941, 996-98 (2002).

[5] Steve Gold, et al., Reference Guide on Epidemiology, at 914, in RMSE 4^th ed.

[6] Id. at 916.

[7] ToxicoGenomica, The Litigator’s Guide to Using Genomics in a Toxic Tort Case (2018).

[8] Id. at 917 & n.55 (citing Bowen v. E.I. Du Pont de Nemours & Co., No. CIV.A. 97C-06-194 CH, 2005 WL 1952859 (Del. Super. Ct. June 23, 2005), aff’d, 906 A.2d 787 (Del. 2006) (discussing the importance of a test for a genetic mutation, which was the defense’s alternative causation theory to plaintiff’s claim that a toxic exposure caused the birth defect at issue). The authors fail to mention that the Bowen case was actually dismissed.

[9] See, e.g., Oliver v. Sec’y Health & Human Servs., 900 F.3d 1357 (Fed. Cir. 2018); Ortega v. United States, 2021 WL 4477896, 2021 U.S. Dist. LEXIS 188969 (N.D.Ill. Sept. 30, 2021); Vanslembrouck ex rel. Braverman v. Halperin, 2014 WL 5462596 (Mich. App. 2014).

[10] See, e.g., Halter v. Boehringer Ingelheim Pharms. Inc., no. 2023-L-001382, Cir. Ct. Cook Cty., Illinois, jury verdict (Aug. 27, 2025) (defense verdict in colorectal cancer case in which plaintiff failed to test for genetic mutation); see also Lauraann Wood, Boehringer Wins Another Zantac Cancer Trial In Illinois, LAW360, Chicago (Aug. 27, 2025).

[11] See, e.g., In re Acetaminophen – ASD-ADHD Prods. Liab. Litig., 707 F.Supp.3d 309, 320 (S.D.N.Y. 2023).

[12] Id. at 916-17 (emphasis added).

[13] Id. at 915.

[14] See, e.g., id. at 967n.190, citing McMillan v. Dep’t of Veterans Affairs, 294 F. Supp. 2d 305, 312 (E.D.N.Y. 2003) (“It is generally accepted that genetic susceptibility plays a key role in determining the adverse effects of environmental chemicals. . . . [I]f polymorphisms of the gene encoding the AhR [protein] exist in humans as they do in laboratory animals, some people would be at greater risk or at lesser risk for the toxic and carcinogenic effects of TCDD [dioxin].”).

[15] See Edward J. Calabrese, Changing the paradigm: The biggest polluter and threat to your health is your body, J. OCCUP. & ENVT’L HYG. (2025), published on-line, ahead of print.

Posted in Causation, Reference Manual on Scientific Evidence | No Comments »

Signature Diseases in the New Reference Manual

January 16th, 2026

The third edition of the Reference Manual on Scientific Evidence[1] had some problems with its discussion of so-called signature diseases.[2] There was a distinct need for the epidemiology chapter in particular to improve in its fourth edition[3] on the issue of so-called signature diseases, diseases caused by only a single cause. The third edition carved out a limited exception to its questionable generalization that epidemiology had nothing useful to say about specific causation by stating that some diseases do not occur without exposure to a specific chemical or substance.

The new, fourth edition carries forward its assertion that “[t]here are some diseases that do not occur without exposure to an agent; these are known as signature diseases.”[4] And in a footnote, the authors of the epidemiology chapter, fourth edition, attempt to explain:

“There are, however, some diseases that do not occur without exposure to a given toxic agent. This is the same as saying that the toxic agent is a necessary cause for the disease, and the disease is sometimes referred to as a signature disease (also, the agent is pathognomonic) because the existence of the disease necessarily implies the causal role of the agent. Two examples are asbestosis, which is a signature disease for asbestos, and vaginal adenocarcinoma (in young adult women), which is a signature disease for in utero DES exposure. See Kenneth S. Abraham & Richard A. Merrill, Scientific Uncertainty in the Courts, in Issues Sci. & Tech. 93, 101 (1986).”[5]

Much of this language in the footnote is repeated from the third edition, as is the citation to the article by Abraham and Merrill. That article was written by lawyers, not scientists, and is now 40 years old, inaccurate and out of date.

With respect to asbestosis, the epidemiology chapter is correct, at least in part. By definition, only asbestos can cause asbestosis, but asbestosis presents clinically in ways that are indistinguishable in many cases from idiopathic pulmonary fibrosis and other interstitial fibrotic diseases of the lungs. Over the years, the diagnostic criteria for asbestosis have changed, but these criteria have always had a specificity and sensitivity less than 100%. Saying that a case of asbestosis must have been caused by asbestos begs the clinical question whether the case really is asbestosis. The situation might be clearer for a pathology diagnosis of asbestosis, but even then there is often the problem of coincidental findings of asbestos bodies in the presence of interstitial fibrosis from another cause.

On the other hand, the chapter’s characterization of vaginal adenocarcinoma as a signature disease of in utero DES exposure is clearly not correct. Although this cancer in young women is rather rare, there is a baseline risk that allows the calculation of relative risks for young women exposed in utero.[6] In older women, the relative risks are lower because the baseline risks are higher, and because the effect of DES is diminished for older onset cases.[7] The disease, however, was known before the use of DES in pregnant women, which began after World War II,[8] and thus not an apt or accurate example of signature disease.

The Reference Manual should really not weigh in on controversies that may arise in courtroom litigations, unless it has a very solid basis. Here the chapter on epidemiology cited to a decades old article, by lawyers, on a technical topic. The proposition about DES was readily falsified by a wee bit of research in PubMed.

[1] National Academies of Sciences, Engineering, and Medicine & Federal Judicial Center, REFERENCE MANUAL ON SCIENTIFIC EVIDENCE (3^rd ed. 2011) (cited as RMSE 3^rd ed.)

[2] See Schachtman, Reference Manual – Desiderata for 4th Edition – Part I – Signature Diseases, TORTINI (Jan. 30, 2023); see also Reference Manual on Scientific Evidence v4.0 (Feb. 28, 2021); Reference Manual on Scientific Evidence – 3rd Edition is Past Its Expiry (Oct. 17, 2021).

[3] National Academies of Sciences, Engineering, and Medicine & Federal Judicial Center, REFERENCE MANUAL ON SCIENTIFIC EVIDENCE (4^th ed. 2025) (cited as RMSE 4^th ed.).

[4] RMSE 4^th ed. at 927-28 n.90.

[5] RMSE 4th at 990 n.274, citing Kenneth S. Abraham & Richard A. Merrill, Scientific Uncertainty in the Courts, 2 ISSUES SCI. & TECH. 93, 101 (Winter 1986). Thankfully, the new epidemiology chapter did not put its finger on the scale about the now discredited view that mesothelioma is a signature disease of asbestos exposure. See Michele Carbone, Harvey Pass, et al., “Medical and Surgical Care of Patients With Mesothelioma and Their Relatives Carrying Germline BAP1 Mutations,” 17 J. THORACIC ONCOL. 873 (2022). See also Mitchell Cheung, et al., Novel LRRK2 mutations and other rare, non-BAP1-related candidate tumor predisposition gene variants in high-risk cancer families with mesothelioma and other tumors, 30 HUMAN MOL. GENETICS 1750 (2021); Thomas Wiesner, et al., “Toward an Improved Definition of the Tumor Spectrum Associated With BAP1 Germline Mutations,” 30 J. CLIN. ONCOL. e337 (2012); Alexandra M. Haugh, et al., Genotypic and Phenotypic Features of BAP1 Cancer Syndrome: A Report of 8 New Families and Review of Cases in the Literature, 153 J.AM. MED. ASS’N DERMATOL. 999 (2017).

[6] See, e.g., Kadir Güzin, et al., “Primary clear cell carcinoma of the vagina that is not related to in utero diethylstilbestrol use,” 3 GYNECOL. SURG. 281 (2006).

[7] Janneke Verloop, et al., Cancer risk in DES daughters, 21 CANCER CAUSES & CONTROL 999 (2010).

[8] See Risk Factors for Vaginal Cancer, American Cancer Soc’y website (last visited Jan. 16, 2026).

Posted in Causation, Reference Manual on Scientific Evidence | No Comments »

Reference Manual 4th Edition Corrects Some, Not All, Mistakes on Confidence Intervals

January 9th, 2026

So now that the new, fourth, edition of the Reference Manual on Scientific Evidence,[1] has been released, inquiring minds may want to know whether it has corrected errors in the previous, third, edition.[2] The authors of the new edition have had 14 years to ponder and reflect upon errors and to correct them.

Judges and lawyers look to the Manual for guidance and understanding of basic concepts, and the first three editions contained significant errors in addressing statistical concepts. There is probably no better place to jump in to see whether the new edition has corrected the prevalent mistakes in defining the statistical concept of a confidence interval, which was botched in several chapters in the third edition.[3] The concept of a confidence interval is important in many statistical applications, but it is especially important in the interpretation of epidemiologic studies.

Contrition is good for the soul. The new edition, in places, evinces an awareness that earlier editions had misled readers, and that the fourth edition needed to do better. And in several key places, including in particular the chapter, the fourth edition has improved in its discussion of confidence intervals.

Professor David Kaye has two chapters in the new edition, one on DNA evidence, and another chapter, with Professor Hal Stern, on statistical evidence.[4] Kaye is a careful writer with substantial statistical expertise. His contributions to the third edition were anodyne treatments of statistical concepts, and his chapters in the new edition seem excellent as well upon first reading. In his chapter on DNA evidence, Kaye alludes to the misunderstandings and misrepresentations of the confidence interval,[5] and in his chapter on statistical evidence, Kaye, along with Stern, gives careful definitions and explications of confidence intervals.

Kaye and Stern call out several cases, frequently cited, for having given clearly incorrect definitions of confidence intervals. This sort of candor to the court is necessary if judges, and lawyers, are going to correct bad practices.[6] The statistics chapter in the fourth edition also does not shy away from calling out the authors of another chapter [epidemiology] in the Reference Manual’s third edition for having given erroneous definitions:

“Language from another reference guide in the previous edition of this Reference Manual that is often quoted may inadvertently convey the incorrect impression that a confidence coefficient such as 95% refers to the percentage of results in (hypothetically) repeated studies that would be expected to lie within the interval reported in the study before the court.”[7]

A very gentle criticism indeed; the epidemiology chapter was manifestly incorrect, and we can all agree that its error was negligent, not intentional. The epidemiology chapter from the third edition did not merely convey the incorrect impression; that chapter contained erroneous definitions of confidence intervals.

Kaye and Sterne correctly note that a given confidence interval “does not give the probability that the unknown parameter lies within the confidence interval.”[8] And they helpfully point out that there is no tendency for the point estimate near the center of a confidence interval to be closer to the true value than any other value within the interval.[9]

The authors of the new edition’s chapter on epidemiology obviously got the message from Professors Kaye and Sterne.[10] Fourth time is a charm. The epidemiology chapter in the third edition had been a mess on statistical issues.[11] Without any acknowledgment or confession of error committed in the first three editions, the authors of the epidemiology chapter in the fourth edition now note:

“Just as the p-value does not provide the probability that the risk estimate found in a study is correct, the confidence interval does not provide the range within which the true risk is likely to lie. In other words, it is a misconception to interpret a 95% confidence interval as representing an interval within which the true value has a 95% probability of being found.”[12]

Unfortunately, in the glossary at the end of the new edition’s epidemiology chapter, the erroneous definition of confidence interval was carried forward from the third edition, without change or correction:

“confidence interval. A range of values that reflects random error. Thus, if a confidence level of 0.95 is selected for a study, 95% of similar studies would result in the true relative risk falling within the confidence interval.”[13]

What the authors no doubt meant to write was that:

“95% of similar studies would result in the true relative risk falling within the confidence intervals.”

By putting “interval” in the singular, the authors fell into the trap described by Professors Kaye and Hall, and into the error that the previous chapters on epidemiology committed.

The new edition of the Reference Manual appears to suffer, at least on this statistical issue, from the lack of high-level editing across chapters. The interaction between authors of the statistics and the epidemiology chapters sorted out a serious error, but the error pops up in new chapters. Michael Weisberg and Anastasia Thanukos have an introductory chapter on How Science Works, which crudely and incorrectly describes confidence intervals:

“Uncertainty and error are generally expressed as a range, within which we are confident that, if the study were repeated, the new result would fall. Scientists often use a 95% confidence interval for this purpose.”[14]

Confidence intervals model only random error, and the “range” around one point estimate does not give us “confidence” that the next point estimate would fall into that range.

The chapter on regression analyses in third edition of the Reference Manual incorrectly defined confidence intervals.[15] Alas the fourth edition did not auto-correct:

“Loosely speaking, a confidence interval represents an interval of values in which the true value of a regression coefficient falls within some pre-specified probability (where the true value is the estimate that would be obtained from the same model with a very large sample).”[16]

Why the authors of a highly technical chapter chose to speak loosely, rather than accurately, is a mystery. All the authors of the regression chapter had to do was refer to the accurate, helpful definitions in the statistics chapter.

Why should we care about the Reference Manual’s misleading, incorrect definitions of confidence intervals (or p-values for that matter)? The erroneous definitions and misuses typically place a Bayesian interpretation upon the confidence interval by claiming that the coefficient of confidence (typically 95% when alpha is set at 0.05) states the probability that the parameter, the true population measure, falls within the interval around the point estimate. This misinterpretation might suffice for a Bayesian 95% credible interval, but almost invariably the calculation under discussion is the point estimate ± 1.96 standard errors. Good statistics, like good grammar, costs nothing.

Whether the conflation of confidence intervals with credible intervals results from ignorance or willful efforts to mislead, it is wrong. And the conflation is part of a long-running rhetorical campaign to mislead about the meaning of the burden of proof and statistical significance in order to abandon statistical tests, and to green-light precautionary principle judgments as “scientific.”[17]

In past posts, I have cited and quoted any number of scientists and lawyers who have engaged in the effort, either intentional or negligent, to mislead readers about the nature of science, by idealizing and falsely elevating the burden of proof in science, and declaring it to be different from the legal and regulatory burden of proof.[18]

To pick one particularly notorious author, consider junk science writer Naomi Oreskes.[19] In her 2010 book, Oreskes declares:

“The 95 percent confidence standard means that there is only 1 chance in 20 that you believe something that isn’t true.

* * * * *

That is a very high bar. It reflects a scientific worldview in which skepticism is a virtue, credulity is not.”[20]

In fact, statistics, science, and law, the confidence interval has nothing to do with the burden of proof; rather it reflects the precision of a single point estimate. Truth is a virtue that may be lost on the likes of Naomi Oreskes, but it is essential to litigating scientific issues. Given that many lawyers in the past had cited the Reference Manual’s chapter on epidemiology for its incorrect definitions of the statistical confidence interval, we should rejoice that this one error has been corrected.

[1] National Academies of Sciences, Engineering, and Medicine & Federal Judicial Center, REFERENCE MANUAL ON SCIENTIFIC EVIDENCE (4^th ed. 2025) (cited as RMSE 4^th ed.)

[2] National Academies of Sciences, Engineering, and Medicine & Federal Judicial Center, REFERENCE MANUAL ON SCIENTIFIC EVIDENCE (3^rd ed. 2011) (cited as RMSE 3^rd ed.)

[3] See Nathan Schachtman, Reference Manual – Desiderata for 4th Edition – Part IV – Confidence Intervals, TORTINI (Feb. 10, 2023).

[4] In RMSE 3^rd ed., Professor Kaye, along with David Freedman, wrote the chapter on statistical evidence; the two gave careful definitions and explications of confidence intervals. Professor Freedman sadly died before the third edition was released, and he is replaced by Hal Stern in the chapter on statistics in the fourth edition.

[5] David H. Kaye, Reference Guide on Human DNA Identification Evidence in RMSE 4^th ed. at 261, (noting that “the meaning of a confidence interval is subtle, and the estimate commonly is misconstrued”).

[6] See Kaye & Sterne, RMSE 4^th ed. at 511 n.125 (citing Turpin v. Merrell Dow Pharm., Inc., 959 F.2d 1349, 1353 (6th Cir. 1992) (“If a confidence interval of ‘95 percent between 0.8 and 3.10 is cited, this means that random repetition of the study should produce, 95 percent of the time, a relative risk somewhere between 0.8 and 3.10.”); Garcia v. Tyson Foods, Inc., 890 F. Supp. 2d 1273, 1285 (D. Kan. 2012) (“Dr. Radwin testified that his study was conducted within a confidence interval of 95 — that is ‘if I did this study over and over again, 95 out of a hundred times I would expect to get an average between that interval.’”); In re Silicone Gel Breast Implants Prods. Liab. Litig., 318 F. Supp. 2d 879, 897 (C.D. Cal. 2004) (“a margin of error between 0.5 and 8.0 at the 95% confidence level . . . means that 95 times out of 100 a study of that type would yield a relative risk value somewhere between 0.5 and 8.0”)).

[7] See Kaye & Sterne, RMSE 4^th ed. at 511 n.125 (citing Rhyne v. U.S. Steel Corp., 474 F. Supp. 3d 733, 744 (W.D.N.C. 2020) (“‘If a 95% confidence interval is specified, the range encompasses the results we would expect 95% of the time if samples for new studies were repeatedly drawn from the population.’ Reference Guide on Epidemiology, at 580.”).

[8] Kaye & Sterne, RMSE 4^th ed. at 512 & n. 126 (citing additional errant judicial decisions, and Geoff Cumming & Robert Maillardet, Confidence Intervals and Replication: Where Will the Next Mean Fall?, 11 PSYCH. METHODS 217 (2006).)

[9] Id. at 512.

[10] Steve C. Gold, Michael D. Green, Jonathan Chevrier, & Brenda Eskenazi, Reference Guide on Epidemiology, in RMSE 4^th ed. at 897

[11] Michael D. Green, D. Michal Freedman & Leon Gordis, Reference Guide on Epidemiology, 549, 573, 580, in RMSE 3^rd ed.

[12] Steve C. Gold, Michael D. Green, Jonathan Chevrier, & Brenda Eskenazi, Reference Guide on Epidemiology, RMSE 4^th ed. at 897, 939.

[13] Id. at 1011.

[14] Michael Weisberg & Anastasia Thanukos, How Science Works , in RMSE 4^th ed. at 47, 90.

[15] Daniel Rubinfeld, Reference Guide on Multiple Regression, RMSE 3^rd ed. at 303, 342, 352.

[16] Daniel Rubinfeld & David Card, Reference Guide on Multiple Regression and Advanced Statistical Models, in RMSE 4^th ed. at 577, 613.

[17] Schachtman, Rhetorical Strategy in Characterizing Scientific Burdens of Proof, TORTINI (Nov. 11, 2014);

[18] See, e.g., Kevin C. Elliott & David B. Resnik, Science, Policy, and the Transparency of Values, 122 ENVT’L HEALTH PERSP. 647 (2014) (exemplifying the rhetorical strategy that idealizes and elevates a burden of proof in science, and then declaring it to be different from legal and regulatory burdens of proof).

[19] Schachtman, Playing Dumb on Statistical Significance, TORTINI (Jan. 4, 2015); The Rhetoric of Playing Dumb on Statistical Significance – Further Comments on Oreskes, TORTINI (Jan. 17, 2015).

[20] Naomi Oreskes & Erik M. Conway, MERCHANTS OF DOUBT: HOW A HANDFUL OF SCIENTISTS OBSCURED THE TRUTH ON ISSUES FROM TOBACCO SMOKE TO GLOBAL WARMING at 156-57 (2010).

Posted in Reference Manual on Scientific Evidence, statistical evidence | No Comments »

A New Year, A New Reference Manual

January 5th, 2026

The fourth edition of the Reference Manual on Scientific Evidence was quietly released in the waning hours of 2025, in the twilight of American democracy.[1] The Manual had been slated to be published in 2023, but that date slid to 2024, and then to 2025. Perhaps the change in directorship of the Federal Judicial Center slowed things up. (Judge Robin Rosenberg of Zantac fame is now the Director)

The new volume is available for download at:

https://www.nationalacademies.org/publications/26919

Although I was a reviewer of one chapter of the Manual, I am just seeing this new edition for the first time today. The basic structure of the volume has not changed, although it has now grown to over 1,600 pages. Many of the key chapters on statistics, epidemiology, toxicology, and medical testimony are carried over from previous editions, with some new authors added and some previous authors no longer participating. In addition, there are some new chapters on exposure science, artificial intelligence, climate science, mental health, neuroscience, and eyewitness identification.

The individual chapters and authors in the new edition of the Manual are:

Liesa L. Richter & Daniel J. Capra, The Admissibility of Expert Testimony, at 1.

Michael Weisberg & Anastasia Thanukos, How Science Works, at 47

Valena E. Beety, Jane Campbell Moriarty, & Andrea L. Roth, Reference Guide on Forensic Feature Comparison Evidence, at 113

David H. Kaye, Reference Guide on Human DNA Identification Evidence, at 207

Thomas D. Albright & Brandon L. Garrett, Reference Guide on Eyewitness Identification, at 361

David H. Kaye & Hal S. Stern, Reference Guide on Statistics and Research Methods, at 463

Daniel L. Rubinfeld & David Card, Reference Guide on Multiple Regression and Advanced Statistical Models, at 577

Shari Seidman Diamond, Matthew Kugler, & James N. Druckman, Reference Guide on Survey Research, at 681

Mark A. Allen, Carlos Brain, & Filipe Lacerda, Reference Guide on Estimation of Economic Damages, at 749

Prologue to the Reference Guide on Exposure Science and Exposure Assessment, the Reference Guide on Epidemiology, and the Reference Guide on Toxicology, at 829i

Elizabeth Marder & Joseph V. Rodricks, Reference Guide on Exposure Science and Exposure Assessment, at 831

Steve C. Gold, Michael D. Green, Jonathan Chevrier, & Brenda Eskenazi, Reference Guide on Epidemiology, at 897

David L. Eaton, Bernard D. Goldstein, & Mary Sue Henifin, Reference Guide on Toxicology, at 1027

John B. Wong, Lawrence O. Gostin, & Oscar A. Cabrera, Reference Guide on Medical Testimony, at 1105

Henry T. Greely & Nita A. Farahany, Reference Guide on Neuroscience, at 1185

Kirk Heilbrun, David DeMatteo, & Paul S. Appelbaum, Reference Guide on Mental Health Evidence, at 1269

Chaouki T. Abdallah, Bert Black, & Edl Schamiloglu, Reference Guide on Engineering, at 1353

Brian N. Levine, Joanne Pasquarelli, & Clay Shields, Reference Guide on Computer Science, at 1409

James E. Baker & Laurie N. Hobart, Reference Guide on Artificial Intelligence, at 1481

Jessica Wentz & Radley Horton, Reference Guide on Climate Science, at 1561

Some quick comments on changes in authorship in some of the chapters. Bernard Goldstein, a member of the dodgy Collegium Ramazzini, remains an author of the toxicology chapter in the new edition. David Eaton, however, has been added. Professor Eaton was the president of the Society of Toxicology for many years, and perhaps he has brought some balance to the new edition’s work on toxicology.

An author of the statistics chapter, David Kaye, is also the sole author of the chapter on DNA evidence. Professor Kaye is a distinguished scholar of DNA evidence with serious statistical expertise. David Freedman had been a co-author of the statistics chapter in the third edition, but sadly Professor Freedman died before the third edition was published. Freedman is replaced by Hal Stern, an accomplished statistician from the University of California.

The chapter on epidemiology lost Leon Gordis, who died in 2015. The chapter in the fourth edition has the return of law professors Steve C. Gold and Michael D. Green, whose pro-plaintiff biases are well known, along with two new authors, epidemiology professors Jonathan Chevrier, & Brenda Eskenazi. Like Goldstein, Eskenazi is a fellow of the Collegium Ramazzini.

The Reference Manual, for better or worse, has had substantial influence on the litigation of scientific and technical issues in federal court, and in some state courts as well. I hope to write more substantively about the new edition in 2026.

[1] National Academies of Sciences, Engineering, and Medicine & Federal Judicial Center, Reference Manual on Scientific Evidence (4^th ed. 2025).

Posted in Causation, Expert Witnesses, Reference Manual on Scientific Evidence, Rule 702 | 1 Comment »

AAAS Conference on Scientific Evidence and the Courts

September 8th, 2025

Back in September 2023, the American Association for the Advancement of Science (AAAS), with its Center for Scientific Responsibility and Justice, sponsored a two day meeting on Scientific Evidence and the Courts. If there were notices for this conference, I missed them. The meeting presentations are now available online. Judging from camera views of the audience, the conference did not appear to be well attended. Most of the material was forgettable, but some of the presentations are worth watching.

Jennifer L. Mnookin opened the conference with a keynote presentation on “Where Law and Science Meet.” Chancellor Mnookin presented a broad overview and some interesting insights on the development of the evidence law of expert witness testimony.

Following Mnookin, Professors Ronald Allen and Andrew Jurs presented on the “Unintended Impacts [sic] of the Daubert Standard.” The conference took place only a few months before amendment to Rule 702 became effective, and the reference to a “Daubert” standard was untoward. Allen’s comments followed the path of his previous articles. Jurs presented some empirical legal research, which seemed flawed for its assumption that the Frye standard was universally applied in federal court before the advent of Daubert. Assessing whether these standards lead to different outcomes when both standards have been applied heterogeneously, and one standard, Frye, is often not applied at all, and Daubert is often flyblown by judges hostile to the gatekeeping enterprise, Jurs’ empirical research seemed both invalid and very much beside the point. Both presenters missed the key point of Daubert, in which case plaintiff’s counsel advocated for no standard at all, beyond basic subject-matter qualification, for giving expert opinions in court.

A Session on “An International Perspective,” Scott Carlson discussed the efforts of the American Bar Association (ABA), and its Center of Global Programs, on supporting judges in foreign countries. Prateek Sibal discussed the history and work of the UNESCO Global Judges Initiative. My sincere wish is that the ABA would support judges more in the United States.

Panelists Valerie P. Hans, Emily Murphy, and Dr. Michael J. Saks presented on various jury issues, in a session “In the Minds of the Jury.” The presentations on how foreign countries process expert witness testimony were lacking any mention of how juries rarely if ever sit in civil cases that involve complex technical and scientific issues.

Two editors of scientific journals, Adriana Bankston and Valda Vinson, along with law professor Michael Sakes, spoke about peer review and publication, in a session “As a Matter of Fact: ‘General Acceptance’ in Emerging vs. Established Science.” Their discussion on the publication process shed very little light on how courts and juries should assess the validity of specific papers, particularly in view of the lax practices at many journals. Towards the end of this session, a question from the audience proved to be very revealing of the prejudices of the law professor on the panel. The questioner rose to complain that after beginning research on a topic that has litigation relevance her research is now frequently questioned. She asked the panel how she might deal with the annoyance of being questioned. Some on the panel basically urged her to buck up, but the law professor invoked the spirit of agnothologist, and lawsuit industry expert witness, David Michael, to suggest that “manufacturing doubt” was just a corporate tactic in the face of scientific evidence. The prejudice against corporate speech is remarkable when the lawsuit industry has a long history of playing the ad hominem game in advancing its pecuniary interests.

The session that followed addressed how trustworthy science might best be put before courts. The organizers described this session, Utilizing Scientific and Technical Expertise, as going to the heart of the issues targeted by the conference. Joe S. Cecil, Deanne M. Ottaviano, and Shari Seidman Diamond discussed how scientific expertise enters into the evidentiary record in American courtrooms. Their presentations were interesting, but curiously no one mentioned that the primary avenue for expert witness opinion is through oral testimony!

Joe Cecil discussed methods judges have to obtain scientific and technical evidence to advance justice. (By this I hope he meant the truth, and not just the outcome preferred by social justice warriors.) As noted, Joe Cecil did not focus on the ordinary methods of direct and cross-examination of party expert witnesses, but rather, he identified other methods of introducing expertise into the courtroom for the benefit of the judge or the jury. Only one suggestion really affects jury comprehension, namely the appointment of non-party expert witnesses by the court. The other methods really only provide expertise to the trial judge, who perhaps is challenged to make a ruling under Federal Rule of Evidence 702. The federal courts have the inherent supervisory power to appoint technical advisors to act as special law clerks on issues. Similarly, appointed special masters can address technical implementation issues, subject to the district judges’ control. The judges are always free to read outside the briefs and testimony, but there are ethical and notice issues for such conduct. The Reference Manual on Scientific Evidence (RMSE) sits on the shelves on every federal judge’s bookshelf, even if in pristine, unused condition. Judges can at least read the RMSE on specific issues without having to disclose their extra-curricular research to the parties. Of course, parties are well advised to consider any materials in the RMSE, which support or oppose their contentions.

In discussing the RMSE, Cecil noted that the fourth edition was in the works. He also mentioned that all the old chapter topics would be carried forward to the fourth edition, and that new topics would include eyewitness identification, computer science, artificial intelligence, and climate science. Sadly, there will be no chapter on genetic determination of disease, but perhaps the clinical medicine chapter will take on the subject in greater detail than previous editions. This conference took place two years ago, and yet the RMSE, fourth edition, is still not published. The National Academies website previously listed the project as completed, but the site now describes the work as “in progress.”

Joe Cecil’s analysis of the various extraordinary expert techniques was pretty much spot on, especially his assessment that “experiments” with court-appointed experts were often failures or at best modest successes. The discussion of Judge Pointer’s Rule 706 independent expert witnesses in the silicon [sic] breast implant litigation, MDL926, seemed to lack context. Cecil acknowledged that the court’s expert witnesses contributed some value to admissibility decisions, but Judge Pointer notoriously did not believe that he, as the MDL judge, had any responsibility for Rule 702 determinations, and he made none except in cases that he tried in the Northern District of Alabama. (And these decisions were before the Science Panel was appointed.) So the Rule 706 witnesses really could not have aided in admissibility decisions.

The real value – in my view – of the Science Panel was that it demonstrated that Judge Pointer was quite wrong in believing that both sides’ expert witnesses were simply “too extreme,” or too partisan, and that the truth was somehow in the middle. Indeed, Judge Pointer said so on many occasions, and he was judicially gobsmacked when all four of his experts roundly rejected the plaintiffs’ distortions of the science of immunology, epidemiology, toxicology, and rheumatology. The courts’ expert witnesses sat for discovery depositions, and then gave testimony de bene esse. To my knowledge, their testimony was never admitted in any of the subsequent trials.

Judge Jed Rakoff gave an interesting presentation, “Strengthening Cooperation Between the Scientific Enterprise and the Justice System,” on the intersection between scientific and legal expertise and the need for their better integration. Judge Rakoff focused on the astonishing lack of compliance of trial judges with the gatekeeping requirements of Rule 702 in addressing the admissibility of forensic evidence. Several subsequent panels also addressed forensic topics, including “A Texas Case Study in Accountability for Forensic Sciences,” “Innovations in Investigative Technologies Improvements and Drawbacks,” and “Artificial Intelligence and the Courts,” “Wrongful Convictions and Changed Science: Statutes,” and “Standing Up for Justice: When the Law and Science Work Hand-in-Hand.”

One of the more curious sessions was on “Statistical Modeling and Causation Science,” presented by the American Statistical Association along with the AAAS. Maria Cuellar, from the University of Pennsylvania, discussed the role of statistical thinking in causal assessment, with slides that referred to a nonparametric estimator for the probability of causation. Cuellar, however, never defined what an estimator was; nor did she differentiate nonparametric from parametric estimators. She displayed other equations, again without explaining their origin and meaning, or identifying symbols or meanings. Similarly, Rochelle E. Tractenberg, discussed the use of statistics as evidence and as part of inferring causal inference in litigation, in a model of unclarity. At one point, Tractenberg appeared to suggest that general causation could be taken from regulatory pronouncements. Her discussion of glyphosate implied that general causation was established, which may have led me to disregard her presentation.

Finally, the conference sported a discussion, “Toxic Tort 2.0: Emerging Trends in Climate Change Related Litigation,” The two presenters were Dr. L. Delta Merner, the “Lead Scientist” for the Science Hub for Climate Litigation, Union of Concerned Scientists, and Dr. Paul A. Hanle, Visiting Scholar and Founder of the Climate Judiciary Project, Environmental Law Institute. The Science Hub actively promotes climate change litigation, which made me wonder whether its scientists are involved in that new chapter in the upcoming fourth edition of the Reference Manual.

Posted in Causation, Reference Manual on Scientific Evidence, Rule 702, Scientific Evidence, Scientific Publishing | 6 Comments »

Systematic Reviews versus Expert Witness Reports

July 2nd, 2025

Back in November 2024, I posted that the fourth edition of the Reference Manual on Scientific Evidence was completed, and that its publication was imminent. I based my prediction upon the National Academies’ website that reported that the project had been completed. Alas, when no Manual was forth coming, I checked back, and the project was, and is as of today, marked as “in progress.” The NASEM website provides no explanation for the retrograde movement. Could the Manual have been DOGE’d? Did Robert F. Kennedy Jr. insist that a chapter on miasma theory be added?

Ever since the third edition of the Manual arrived, I have tried to identify its strengths and weaknesses, and to highlight topics and coverage that should be improved in the next edition. In 2023, knowing that people were working on submissions for the fourth edition, I posted a series of desiderata for the new edition.[1] I might well have extended the desiderata, but I thought that work was close to completion.

One gaping omission in the third edition of the Manual, which I did not address, is the dearth of coverage of the synthesis of data and evidence across studies. To be sure, the chapter on medical testimony does discuss the “hierarchy of medical evidence, and places the systematic review at the apex.[2] The chapter on epidemiology, however, fails to discuss systematic reviews in a meaningful way, and treats meta-analysis, which ideally pre-supposes a systematic review, with some hostility and neglect.[3]

Notwithstanding the glaring omission in the 2011 version of the Reference Manual, the legal academy had been otherwise well aware of the importance of properly conducted systematic reviews. Back in 2006, Professor Margaret Berger organized a symposia on law and science, at which John Ioannidis presented on the importance of systematic reviews.[4] Lisa Bero also presented on systematic reviews and meta-analyses, and identified a significant source of bias in such reviews that results when authors limit their citations to studies that support their pre-selected, preferred conclusion.[5] Bero’s contribution, however, missed the point that a well-conducted systematic review makes cherry picking much more difficult, as well as obvious to the reader.

The high prevalence of biased citation and consideration of, and reliance upon, studies is a major source of methodological error in courtroom proceedings. Even when the studies relied upon are reasonably well done, expert witnesses can manipulate the evidentiary display through biased selection and exclusion of what to present in support of their opinions. Sometimes astute judges recognize and bar expert witnesses who would pass off their opinions, as well considered, when they are propped up only by biased citation. Unfortunately, courts have not always been vigilant and willing to exclude expert witnesses who proffer biased, invalid opinions based upon cherry-picked evidence.[6] Given that cherry picking or “biased citation” is recognized in the professional community as rather serious methodological sins, judges may be astonished to learn that both phrases, “cherry picking” and “biased citation” do not appear in the third edition of the Reference Manual on Scientific Evidence. With the delay in publishing the fourth edition, there is still time to add citations to careful debunking of biased citation, such as the reverse-engineered systematic review and meta-analysis in last year’s decision in the paraquat parkinsonism litigation.[7]

When I began my courtroom career, systematic reviews of the evidence for a causal claim were virtually non-existent. Most reviews and textbook chapters were hipshots that identified a few studies that supported the author’s preferred opinion, with perhaps a few disparaging words about a study that contradicted the author’s preferred outcome. On a controversial issue, lawyers could generally find a textbook or review article on either side of an issue. Cross-examination on a so-called “learned treatise,” however, was limited. In state courts, the learned treatise was not admissible for its truth, but only to show that expert witnesses should not be believed when they disagreed with the statement. It was all too easy for an expert witness to declare, “yes, I disagree with that one sentence, on one page, out of 1,500 pages, in that one book.”

In federal courts, the applicable rule of evidence makes the learned treatise statement admissible for its truth:

“Rule 803. Exceptions to the Rule Against Hearsay

The following are not excluded by the rule against hearsay, regardless of whether the declarant is available as a witness:

(18) Statements in Learned Treatises, Periodicals, or Pamphlets . A statement contained in a treatise, periodical, or pamphlet if:

(A) the statement is called to the attention of an expert witness on cross-examination or relied on by the expert on direct examination; and

(B) the publication is established as a reliable authority by the expert’s admission or testimony, by another expert’s testimony, or by judicial notice.

If admitted, the statement may be read into evidence but not received as an exhibit.”

While this rule historically had some importance in showing the finder of fact that the opinion given in court was not shared with the relevant expert community, the rule was and is problematic. Exactly what counts as “learned” is undefined. Expert witnesses on either side can simply endorse a treatise, a periodical, or a pamphlet as learned to enable a lawyer to use it on direct or cross-examination, and make its contents admissible. The rule was drafted and enacted in 1975, when another rule, Rule 702, was generally interpreted to place no epistemic restraints upon expert witnesses. Allowing Rule 803(18) to be invoked without the epistemic constraints of Rules 702 and 703 raised few concerns in 1975, but in the aftermath of Daubert (1993), the tension within the Federal Rules of Evidence requires that the admissibility of a statement in a learned treatise cannot save an expert witness opinion that is not otherwise sufficiently grounded and valid.[8]

Systematic reviews are a different kettle of fish from the sort of textbook opinions of the 1970s and 1980s, which often lacked comprehensive assessments and consistent application of criteria for validity. The intersection of the evolution of Rule 702 and systematic reviews is remarkable. When Rule 702 was drafted, systematic reviews were non-existent. When the Supreme Court decided the Daubert case in 1993, systematic reviews were just emerging as a different and superior form of evidence synthesis.[9] The lesson for judges, regulators, and lawyers is that the standards for valid synthesis of studies and lines of evidence have changed and become more demanding.

In 2009, several professional groups produced an important guidance for reporting systematic reviews, “the Preferred Reporting Items for Systematic reviews and Meta-Analyses,” or PRISMA.[10] Although the PRISMA guidance ostensibly addresses reporting, if authors have not done something that should be reported, their failure to do it and report about it can be identified as a significant omission from their publication. One of the PRISMA specifications called for the writing of a protocol for any systematic review, and for making this protocol available to the scientific community and the public. The protocol will identify the exact clinical issue under review, the kinds of evidence that bear on the issue, and criteria for including or excluding studies that should be included in the review. The requirement of pre-registration has the ability to damp down data dredging in observational studies and experiments, and to help readers see when authors reverse engineered systematic reviews by declaring their criteria for inclusion and exclusion after reading candidate studies and their conclusions.

In 2011, the Centre for Reviews and Dissemination, at the University of York in England, developed an internet archive, PROSPERO, for prospectively registering systematic reviews. In addition to reducing duplication of systematic reviews, PROSPERO aimed to increase transparency, validity, and integrity of the systematic reviews. Around the same time, the Center for Open Science, also set up a web-based archive for systematic review protocols.[11]

Reviews purporting to be systematic are now commonplace. By 2018, ROSPERO had registered over 30,000 records, but of course, some scientists may have registered systematic reviews which they never completed.[12] Despite the publication of professional guidances, carefully performed systematic reviews can still be hard to find.[13]

In federal court, expert witnesses must proffer their opinions in a specified form. Back in the 1980s, federal court practice on expert witnesses was “loose” not only on admissibility issues, but also on the requirements for pre-trial disclosure of opinions. In some federal districts, such as those within Pennsylvania, federal judges took their cues not from the language of the Federal Rules of Civil Procedure, but from state court practice, which required only cursory disclosure of top-level opinions without identifying all facts and data relied upon by the proposed expert witness. In many state courts, and in some federal judicial districts, lawyers had a difficulty obtaining judicial authorization to conduct examinations before trial to discover all the bases and reasoning (if any) behind an expert witness’s opinion. Under the current version of the Federal Rules of Civil Procedure, trial by ambush has generally given way to full discovery. The current version of Rule 26 provides:

Rule 26. Duty to Disclose; General Provisions Governing Discovery

(a) Required Disclosures.

* * *

(2) Disclosure of Expert Testimony.

(A) In General. In addition to the disclosures required by Rule 26(a)(1) , a party must disclose to the other parties the identity of any witness it may use at trial to present evidence under Federal Rule of Evidence 702 , 703 , or 705 .

(B) Witnesses Who Must Provide a Written Report. Unless otherwise stipulated or ordered by the court, this disclosure must be accompanied by a written report—prepared and signed by the witness—if the witness is one retained or specially employed to provide expert testimony in the case or one whose duties as the party’s employee regularly involve giving expert testimony. The report must contain:

(i) a complete statement of all opinions the witness will express and the basis and reasons for them;

(ii) the facts or data considered by the witness in forming them;

(iii) any exhibits that will be used to summarize or support them;

(iv) the witness’s qualifications, including a list of all publications authored in the previous 10 years;

(v) a list of all other cases in which, during the previous 4 years, the witness testified as an expert at trial or by deposition; and

(vi) a statement of the compensation to be paid for the study and testimony in the case.

An expert’s report or disclosure under Rule 26 remains a far cry from a systematic review, but the Rule goes a long way towards eliminating trial by ambush and surprise in requiring a complete statement of all opinions, all the bases and reasons for the opinions, and all the facts or data considered in reaching the opinions. The requirements of Rule 26, combined with a mandatory oral deposition, go a long way to help reveal cherry picking and motivated reasoning in an expert witness’s opinions.

[1] Schachtman, “Reference Manual – Desiderata for 4th Edition – Part I – Signature Diseases,” Tortini (Jan. 30, 2023); “Reference Manual – Desiderata for 4th Edition – Part II – Epidemiology & Specific Causation,” Tortini (Jan. 31, 2023); “Reference Manual – Desiderata for 4th Edition – Part III – Differential Etiology,” Tortini (Feb. 1, 2023); “Reference Manual – Desiderata for 4th Edition – Part IV – Confidence Intervals,” Tortini (Feb. 10, 2023); “Reference Manual – Desiderata for 4th Edition – Part V – Specific Tortogens,” Tortini (Feb. 14, 2023); “Reference Manual – Desiderata for 4th Edition – Part VI – Rule 703,” Tortini (Feb. 17, 2023).

[2] See John B. Wong, Lawrence O. Gostin, and Oscar A. Cabrera, “Reference Guide on Medical Testimony,” in Reference Manual on Scientific Evidence 687, 723-24 (3d ed. 2011) (discussing hierarchy of medical evidence, with systematic reviews at the apex).

[3] Schachtman, “The Treatment of Meta-Analysis in the Third Edition of the Reference Manual on Scientific Evidence,” Tortini (Nov. 14, 2011).

[4] John P.A. Ioannidis & Joseph Lau, Systematic Review of Medical Evidence, 12 J.L. & Pol’y 509 (2004).

[5] Lisa Bero, “Evaluating Systematic Reviews and Meta-Analyses,” 14 J. L. & Policy 569, 576 (2006).

[6] See Schachtman, “Cherry Picking; Systematic Reviews; Weight of the Evidence,” Tortini (April 5, 2015); “The Fallacy of Cherry Picking As Seen in American Courtrooms,” Tortini (May 3, 2014); “The Cherry-Picking Fallacy in Synthesizing Evidence,” Tortini (June 15, 2012).

[7] In re Paraquat Prods. Liab. Litig., 730 F. Supp. 3d 793 (S.D. Ill. 2024); see also Schachtman, “Paraquat Shape-Shifting Expert Witness Quashed,” Tortini (Apr. 24, 2024).

[8] See Schachtman, “Unlearning the Learned Treatise Exception,” Tortini (Aug. 21, 2010).

[9] Iain Chalmers, Larry V. Hedges, Harris Cooper, “A Brief History of Research Synthesis,” 25 Evaluation & the Health Professions 12 (2002); Mark Starr, Iain Chalmers, Mike Clarke, Andrew D. Oxman, “The origins, evolution, and future of The Cochrane Database of Systematic Reviews,” 25 Int J. Technol. Assess. Health Care s182 (2009); Mike Clarke, “History of evidence synthesis to assess treatment effects: personal reflections on something that is very much alive,” 109 J. Roy. Soc. Med. 154 (2016). See also Wen-Lin Lee, R. Barker Bausell & Brian M. Berman, “The growth of health-related meta-analyses published from 1980 to 2000,” 24 Eval. Health Prof. 327 (2001).

[10] Alessandro Liberati, Douglas G. Altman, Jennifer Tetzlaff, Cynthia Mulrow, Peter C. Gøtzsche, John P.A. Ioannidis, Mike Clarke, Devereaux, Jos Kleijnen, and David Moher, “The PRISMA Statement for Reporting Systematic Reviews and Meta-Analyses of Studies That Evaluate Health Care Interventions: Explanation and Elaboration,” 151 Ann Intern Med. W-65 (2009); “The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration,” 6 PLoS Med. e1000100 (2009).

[11] Alison Booth, Mike Clarke, Gordon Dooley, Davina Ghersi, David Moher, Mark Petticrew & Lesley Stewart, “The nuts and bolts of PROSPERO: an international prospective register of systematic reviews,” 1 Systematic Reviews 1 (2012); Alison Booth, Mike Clarke, Davina Ghersi, David Moher, Mark Petticrew, Lesley Stewart, “An international registry of systematic review protocols,” 377 Lancet 108 (2011).

[12] Matthew J. Page, Larissa Shamseer, and Andrea C. Tricco, “Registration of systematic reviews in PROSPERO: 30,000 records and counting,” 7 Systematic Reviews 32 (2018).

[13][13] John P. Ioannidis, “The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-analyses,” 94 Milbank Q. 485 (2016).

Posted in Expert Witness Discovery, Expert Witnesses, Reference Manual on Scientific Evidence, Rule 702, Scientific Publishing | No Comments »

Science for Judges – Reference Manual v4.0

November 6th, 2024

By the time the third edition of the Reference Manual on Scientific Evidence (RMSE) arrived in 2011, the work had evolved into a massive doorstop. The third edition generally got favorable, but unsearching, reviews. In some ways it was an impressive effort, but it left a lot to be desired in terms of comprehensiveness and consistency.[1] A decade passed, and the National Academies of Science, Engineering, and Medicine (NASEM), along with the Federal Judicial Center, opened work on a fourth edition, in early 2021.[2]

A look at the NASEM website shows that work on the fourth edition of the RMSE is now completed. There is, however, no announced publication date. The website’s description of the RMSE project suggests that the fourth edition will continue the practice of individual chapters with different authors. The topics to be covered are listed as:

Behavioral and Social Sciences, Biology and Life Sciences, Computers and Information Technology, Earth Sciences, Education, Engineering and Technology, Environment and Environmental Studies, Health and Medicine, Math, Chemistry, and Physics, Policy for Science and Technology, and Surveys and Statistics.

It seems unlikely that the chapters will actually track these topics. Previous editions had specific chapters on epidemiology, toxicology, regression, and clinical medicine, among others. The listing of topics strikes me as a higher level of generality than the actual chapter headings.

The following project description is provided:

“In collaboration with the Federal Judicial Center (FJC), a committee of the National Academies of Sciences, Engineering, and Medicine will develop the fourth edition of the Reference Manual on Scientific Evidence. The Reference Manual is a primary reference source for federal judges on questions of science in litigation. It does not instruct judges on how to rule regarding admissibility of particular types of evidence, but instead offers judges advice on how to manage expert testimony, discusses emerging problems with expert testimony, and provides information on the methodology of areas of science that often present difficult issues when introduced in the form of expert testimony.

The manual is a compilation of individually-authored chapters on various topics of science and technology relevant to the courts, The fourth edition will include updates of existing chapters as well as new chapters that reflect emerging areas. The committee will select the topics to be included in the manual, commission expert authors to revise the current chapters or draft new ones, approve the chapters, and submit the manual for external review.”

This description, at least as to previous editions, seems misleading. The first, second, and third editions contained very specific advice on specific issues. Indeed, it is unfathomable how a reference manual could avoid prescriptive judgments as to how scientific judgments should and should not be reached.

The Co-Chairs of the fourth edition are Hon. Nancy D. Freudenthal and Dr. Fred H. Gage. Members of the committee responsible for the new edition are:

Dr. Russ B. Altman (biomedical data, pharmacogenomics)

Hon. David G. Campbell (D. Ariz.)

Dr. Alicia L. Carriquiry (statistics, forensics)

Dr. Lynn R. Goldman (occupational and environmental health)

Dr. Brian W. Kernighan (engineering)

Dr. Pramod P. Khargonekar (engineering)

Hon. Goodwin Liu (California Supreme Court)

Dr. Shobita Parthasarathy (science, technology, and public policy)

Hon. Patti B. Saris (D. Mass.)

Hon. Thomas Schroeder (M.D.N.C.)

Hon. David S. Tatel (6^th Circuit)

The Staff Officer for the project is Dr. Anne-Marie C. Mazza.

There is much that is needed in a new edition. We will soon know whether the wait was worth it.[3]

[1] See, e.g., Adam Dutkiewicz, “Book Review: Reference Manual on Scientific Evidence, Third Edition,” 28 Thomas M. Cooley L. Rev. 343 (2011); John A. Budny, “Book Review: Reference Manual on Scientific Evidence, Third Edition,” 31 Internat’l J. Toxicol. 95 (2012); James F. Rogers, Jim Shelson, and Jessalyn H. Zeigler, “Changes in the Reference Manual on Scientific Evidence (Third Edition),” Internat’l Ass’n Def. Csl. Drug, Device & Biotech. Comm. Newsltr. (June 2012). See Schachtman “New Reference Manual’s Uneven Treatment of Conflicts of Interest,” Tortini (Oct. 12, 2011).

[2] Schachtman,“Reference Manual on Scientific Evidence v4.0” Tortini (Feb. 28, 2021); Schachtman, “People Get Ready – There’s A Reference Manual A’Comin’,” Tortini (July 16, 2021); “Reference Manual on Scientific Evidence – 3rd Edition is Past Its Expiry” Tortini (Oct. 17, 2021).

[3] I have written elsewhere of some of the issues that cry out for attention. Schachtman, “Reference Manual – Desiderata for the 4^th Edition – Part I – Signature Diseases,” Tortini (Jan. 30, 2023); “Reference Manual – Desiderata for the 4^th Edition – Part II – Epidemiology and Specific Causation,” Tortini (Jan. 31, 2023); “Reference Manual – Desiderata for the 4th Edition – Part III – Differential Diagnosis,” Tortini (Feb. 1, 2023); “Reference Manual – Desiderata for the 4th Edition – Part IV – Confidence Intervals,” Tortini (Feb. 10, 2023); “Reference Manual – Desiderata for the 4th Edition – Part V – Specific Tortogens,” Tortini (Feb. 14, 2023); “Reference Manual – Desiderata for the 4th Edition – Part VI – Rule 703,” Tortini (Feb. 17, 2023).

Posted in Reference Manual on Scientific Evidence | 10 Comments »