Schachtman Law » Scientific Evidence

TORTINI

For your delectation and delight, desultory dicta on the law of delicts.

Kiker v. Smithkline Beecham & the Pathology of Judicial Gatekeeping

January 4th, 2017

“There is no expedient to which a man will not go to avoid the labor of thinking.” Sir Joshua Reynolds

Medical students study pathology not only to understand the nature, course, and causation of disease, but also to understand better normal tissue and cellular function and structure. Similarly, lawyers can improve their understanding of judicial decision making, not only from studying well-reasoned judicial opinions, but from also studying pathological opinions, with clear, demonstrable errors that help illustrate both the pathogenesis of intellectual and judicial error, as well as the normal, proper function of judging.

At the end of each year, bloggers and pundits traditionally call attention to the best and the worst decisions, usually from a partisan perspective. One federal judicial decision on Rule 702, however, stands out for special treatment as a veritable Berenstain Bears’ manual on how not to adjudicate so-called Daubert motions. Kiker v. Smithkline Beecham Corp., 2:14-cv-02164-EAS-TPK, (S.D. Ohio, Dec. 15, 2016) (Sairgus, C.J.) [cited below as Kiker slip op.] The Kiker opinion is as worthy of dissection as a judicial opinion for lawyers, as is the dissection of a cadaver by medical students in their first-year course on clinical anatomy.

The Kiker plaintiffs claimed that maternal use of paroxetine (tradename Paxil) caused her child to develop a ventricular septal defect. The defendant, GlaxoSmithKline LLC (GSK), invoking Federal Rule of Evidence 702, moved to exclude opinion testimony of several of plaintiffs’ expert witnesses, including Laura M. Plunkett, Ph.D., Ra-id Abdulla, M.D. Kiker slip op. at 1. The gravaman of the plaintiffs’ case is that GSK did adequately warn physicians of the risk to offspring of women who took paroxetine in pregnancy until September 2005. At that time, GSK revised its labeling for Paxil to warn of the “increased risk for cardiovascular malformations.” Kiker slip op. at 3.

The plaintiffs threw in the kitchen sink with their allegations, which included specific averments that GSK should have informed the medical community about “significant” adverse event reporting and the meaning of claimed deaths among rat pups in high-dose maternal toxicity testing. Not content with a failure to warn case, plaintiffs ratcheted their allegations into a fraudulent misrepresentation case, as well. Kiker slip op. at 3-4. Laura Plunkett and Ra-id Abdulla were the principal expert witnesses relied upon by plaintiffs for their hyperbolic claims.

The Standard

Chief Judge Sargus started his description of the governing law by insisting that the standard for expert witness gatekeeping was “flexible”; that is, he would follow the “Gumby Rule,” which allows the trial judge maximal flexibility and stretch to admit dubious expert witness opinions. Chief Judge Sargus employed the usual reductionist criteria for assessing “reliability.” Citing Kumho Tire, he explained that the court’s role was to ascertain whether

“an expert . . . employs in the courtroom the same level of intellectual rigor that characterizes the practice of an expert in the relevant field.”

Kumho Tire Co. v. Carmichael, 526 U.S. 137, 152, (1999). He also acknowledged that Daubert had provided some indicia of reliability in factors such as

“testing, peer review, publication, error rates, the existence and maintenance of standards controlling the technique’s operation, and general acceptance in the relevant scientific community.”

Kiker slip op. at 7, quoting from United States v. Langan, 263 F.3d 613, 621 (6th Cir. 2001) (citing Daubert v. Merrell Dow Pharm, Inc., 509 US. 579, 593-94 (1993)).

Chief Judge Sargus was then quick to point out that the cited Daubert factors do not make up a definitive, dispositive test or checklist, which presumably gave him license to ignore these factors and their absence, all together. Nowhere later in his opinion on the contested reliability of plaintiffs’ expert witnesses’s causation opinions is there any discussion of the actual testing, its validity, its pre-publication and post-publication peer review, error rates, standards for assessing causation, or general acceptance of the claimed methodologies. And of course, the discretion permitted district judges in performing their gatekeeping function is not the discretion to abandon the gatekeeping function and to ignore relevant methodological criteria. See Kumho Tire, 526 U.S. 137, 158-59 (Scalia, J., concurring).

Semantic Legerdemain Substitutes for Demonstration of General Causation

Chief Judge Sargus acknowledged that there is a “specific methodology” used by scientists to assess a body of evidence for causation of birth defects, but then proceeded to ignore that methodology without bothering to describe or apply it. Kiker slip op. at 10. What gave the trial judge his argument for ignoring the “specific methodology” used by scientists, the Daubert factors, and indeed any and all factors for assessing the validity of a scientific claim and conclusion, was the language used by the Food and Drug Administration (FDA) and GSK, the NDA-holder, in various communications. Rather than engage in an intellectually challenging exploration and evaluation of the actual scientific evidence and analysis that underlay the plaintiffs’ expert witnesses’ causation opinions, Chief Judge Sargus pointed to the language used by the FDA in its original Public Health Advisory about the issue of congenital cardiac malformations in children of mothers who ingested paroxetine in their first trimester of pregnancy:

“[t]he FDA has determined that exposure to paroxetine in the first trimester of pregnancy may increase the risk for congenital malformations, particularly cardiac malformations. At the FDA’s request, the manufacturer has changed paroxetine’s pregnancy category from C to D and added new data and recommendations to the WARNINGS section of paroxetine’s prescribing information. FDA is awaiting the final results of recent studies and accruing additional data related to the use of paroxetine in pregnancy in order to better characterize the risk for congenital malformations associated with paroxetine.”

Kiker slip op. at 10, quoting from FDA Public Health Advisory (Dec. 8, 2005), available at <http://www.fda.gov/Drugs/DrugSafety/PostmarketDrugSafetyInformationforPatientsandProviders/ucm051731.htm> (emphasis added).

Chief Judge Sargus apparently was oblivious to the difference between “X causes Y” and “X may increase the risk of Y.” As the trial judge, he also fixed on the FDA’s decision to change the pregnancy category labeling for paroxetine from Category C to Category D, with the latter category’s reflecting “positive evidence of human risk.” Kiker slip op. at 11. Again, the existence of evidence for risk is not, and never has been, the existence of evidence that would support a reasonable, reliable conclusion that paroxetine causes cardiac birth defects. Nothing can explain or justify this incredible reliance and misinterpretation of language, and Chief Judge Sargus makes no attempt to defend his linguistic contortions.

Chief Judge Sargus ends with an implied assertion that he, as trial judge, need not spend any time on assessing the quantity or quality of evidence for a conclusion of causality because GSK has admitted that paroxetine causes cardiac birth defects. The GSK Dear Healthcare Provider Letter, the FDA Safety Alert, along with the (preliminary) results of a single epidemiologic study

“combine in this instance to constitute an admission that Paxil can cause injury, and is sufficient to create an issue of fact regarding causation.”

Kiker slip op. at at 15.

Whence comes this incredible reliance upon the language of a package insert? Chief Judge Sargus points to Judge James Gwin’s decision in In re Meridia, and proceeds to provide two pages, single-spaced, of block quotation from the Meridia decision. Kiker slip op. at 13-15, quoting from In re Meridia Prods. Liab. Litig., 328 F. Supp. 2d 791, 800-01 (N.D. Ohio 2004).

Interspersed in the two pages of quotation from Meridia were citations to Ferebee and Wells, two of the most discredited, disreputable federal court decisions on biomedical causation, both of which were effectively overruled sub silentio by the Supreme Court in Daubert. Chief Judge Sargus argues that the Meridia decision held that “product inserts to both physicians and patients” constituted “admissions of Meridia’s potential to cause substantial increases in blood pressure in some patients. Meridia, 328 F. Supp. 2d at 810. Affirming the district court’s decision in Meridia, the Sixth Circuit specifically upheld the district court’s determination that the FDA warning label at issue in that case “constitutes an admission that Meridia can cause injury.” Meridia Prods. Liab. Litig. v. Abbott Labs, 447 F.3d 861, 866 (6th Cir. 2006).

This analytical shortcut has serious problems. First, as a first year law student might observe, the Meridia decision resulted in the exclusion of plaintiffs’ key expert witness and the grant of summary judgment to the defendant on adequacy of its warning, all of which the Sixth Circuit affirmed. Given that there was no liability, the comments about causation would seem to be dictum, not holding. Second, with respect to the issue of warnings as admissions, the Circuit agreed that the district court had construed the defendant’s package insert warning that the medication ‘‘substantially increases’’ blood pressure as an admission, but that such unequivocal language was quite different from warning language that states medication use ‘‘is associated with’’ an adverse event. 447 F.3d at 866. The FDA’s Public Health Advisory, the change to Category D, and GSK’s own sponsored study did not, individually or collectively, state a finding of anything more than an association, and that there “may be an increased risk.”

Of course, Chief Judge Sargus’s glib exercise eliminated all the difficult thought of evaluating actual scientific evidence. The indolent approach used in Kiker committed another blatant error. The approach not only relied incorrectly upon some language of the FDA and medication license holder, but it ignored all the contrary evidence, context, and analysis that kept the FDA from reaching a conclusion of causality in 2005, and most scientists to this very day. Furthermore, the Kiker approach conveniently ignored that over a decade of additional evidence, much of it exonerating paroxetine. Chief Judge Sargus has misidentified the weakest, incomplete, out-of-date, cherry-picked evidentiary display with reliable evidence that purports to support a causal conclusion.

Non-Specific Confusion on Specific Causation

Having announced that the court will not grant a hearing, or even an on-the-paper review of the actual evidence for plaintiffs’ causal claims, Chief Judge Sargus proceeded to make even shorter work of the issue of specific causation. The only support for specific causation in the case was in the proffered testimony of Dr. Ra-id Abdulla, a serial testifying expert witness in anti-depressant birth defects cases. Abdulla purported to conduct a differential diagnosis to discern the cause of the infant plaintiff’s birth defect, a ventricular septal defect. Kiker slip op. at 16.

The diagnosis of the infant Kiker’s birth defect, however, was never in doubt; rather it was the etiology of the septal defect, which was at issue. Abdulla claimed to have ruled out all other potential alternative causes. Kiker slip op. at 18. Even if Abdulla’s claim could be accepted for known causes of septal defects, he would still be faced with a situation in which there are baseline or background cases of septal defects, which occur in children with no known or even suspected risk factor. The court failed to explain how Abdulla ruled out such unknown, prevalent causes of septal defects in the Kiker plaintiff. To be sure, the court appeared to have fallen for the “treating physician” ruse, which suggests that treating a condition provides some magical insight into the cause of that condition. Kiker slip op. at 19-20.

No explanation was cited by the court for how Abdulla worked his magical clinical inference of specific causation. Sadly, there is no such magic, except in the form of the magic thinking evidenced here by Abdulla, and acquiesced in by Chief Judge Sargus. No biomarker of causal originst distinguishes the Kiker plaintiff’s septal defect from one caused by any other cause, whether or not established by current medical science. Moreover, Abdulla’s magical thinking cannot be swept under the Kumho Tire rug of appropriate level of rigor in the field. The Kiker court cited no evidence that pediatric cardiologists routinely and reliably make the specific causal attribution that Dr. Abdulla made in this case, as a paid, testifying expert witness. The court incredulously accepted Abdulla’s hand waving about the epistemic warrant of experience, education, training that has nothing to do with discerning individual causes.

GSK asked for oral argument, which may have been Chief Judge Sargus’s last clear chance to avoid these errors. Declaring that the record was fully developed, Judge Sargus denied the request for a hearing. Kiker slip op. at 1, 4. We are left with a profoundly flawed misunderstanding of scientific evidence and causal inference.

Posted in Causation, Rule 702, Scientific Evidence | Comments Off on Kiker v. Smithkline Beecham & the Pathology of Judicial Gatekeeping

Fake Science News

December 17th, 2016

Fakers to the left; fakers to the right. Everyone has his or her knickers in a knot over fake news these days. But who will speak out against fake science news?

Oberle Communications LLC [1] puts out a “Product Safety Letter,” with almost daily emails that link to published articles of interest to lawyers and others who are concerned with product safety. According to its self-description, Product Safety Daily is a “fair-use news-link service,” and its owner claims not to be responsible for the accuracy, or truthfulness, of linked articles.

Sounds like Facebook; no?

The Guardian is British newspaper, with affiliates in the United States and elsewhere, owned by the Guardian Media Group, which in turn is owned by The Scott Trust Limited. The Scott Trust declares that it exists to produce The Guardian, and “to safeguard the journalistic freedom and liberal values of The Guardian free from commercial or political interference.” Lofty goals, those are. Oberle Communications might feel secure in pointing to an article in The Guardian, on product safety. As far as newspapers are concerned, The Guardian enjoys a good reputation, and has won awards for its investigative journalism, most recently on unlawful government surveillance.

Recently, the Product Safety Letter linked to an article by an Assistant Editor of The Guardian on supposed health effects of plastics. Chukwuma Muanya, “How Plastics Cause Autism, Diabetes, Cancer, Birth Defects,” The Guardian (London, England) (Dec. 13, 2016).

The Mr. Muanya’s headline shouts about causation, but there is nothing in the text of the article to support, even remotely, anyone’s conclusion about causality. The text of the article states, without support, that “[r]ecent studies have associated the rise in autism, diabetes, cancer and birth defects to increase in the use of plastics in making everyday containers, toys and baby teethers or pacifiers.” One would think, hope, pray that The Guardian would know the difference between association and causation, but there is no evidence in this article to support an imputation of knowledge or understanding.

A photograph of baby bottles contains a caption that ramps up the Guardian’s rhetoric and propaganda:

“KILLER PLASTICS… The invisible chemical cause neurological and behavioral disorders like autism and attention deficit and hyperactivity disorder (ADHD). They also affect IQ. And they manipulate hormones in a way that can cause cancer, diabetes, male infertility, and endometriosis. PHOTO CREDIT: http://www.viewzone.com/plastic-header.jpg”

Wow. These plastics are bad-ass actors. They manipulate; they cause; they kill.

From causation, to association, the author moves to the most abused journalese term in science reportage: “link”:

“the presence of toxins commonly found in plastic that have been linked to increased risk of cancer, heart disease and obesity.”

Mr. Muanya references, without link, citation, or mention of authors, a study by the American Chemical Society, which apparently reported that infants’ pacifiers contained Bisphenol A (BPA), Bisphenol S (BPS) or Bisephenol F (BPF), and that many also contained parabens, and antimicrobials such as triclosan and triclocarban. But the Society’s paper was about chemical content, not about health consequences. Without any reference or citation to published or unpublished studies, Mr. Muanya labels BPA, BPS, and BPF as “so-called endocrine-disrupting chemicals,” and tells us these chemicals “manipulate hormones in a way that can cause cancer, diabetes, male infertility, and endometriosis,” and that these chemicals “cause neurological and behavioral disorders like autism and ADHD,” and that they “also affect IQ.”

Apropos of nothing having to do with endocrine disrupters, or human disease, Mr. Muanya inserts a discussion of a Japanese study, reportedly published at the PLoS (but without providing link or citation) about how older male mice have offspring that exhibited “hyperlocomotion.” Still, Mr. Muanya, who has been billed as “Head Insight Team, Science & Technology,” does show that journalists can provide, when it suits them, actual references at least to authors by name. Nothing in the mouse study, however, explains the hypocognition exhibited by the Guardian’s science editor, or the shoddy journalistic practices.

So here we have a respected newspaper publishing a news story that at best is internally inconsistent and un-sourced, and which grossly misinterprets or overinterprets the available scientific evidence. Behold fake science news.

[1] 4915 St Elmo Ave, #204, Bethesda, MD 20814; Phone: (301) 215-9236.

Posted in Scientific Evidence | Comments Off on Fake Science News

Talc Litigation – Stop the Madness

November 10th, 2016

Back in September, Judge Johnson, of New Jersey, wrapped up a talc ovarian cancer case in Kemp, and politely excused the case from any further obligations to show up in court. Carl v. Johnson & Johnson, No. ATL-L-6546-14, 2016 WL 4580145 (N.J. Super. Ct. Law Div., Atl. Cty., Sept. 2, 2016) [cited as Carl]. See “New Jersey Kemps Ovarian Cancer – Talc Cases” (Sept. 16, 2016).

In Giannecchini v. Johnson & Johnson, a Missouri jury returned a substantial verdict for plaintiff. The jury, by a 9 to 3 vote, awarded $575,000 for claimed economic loss, and $2 million for non-economic compensatory damages. The jury also found defendant Johnson & Johnson in need of punishment to the tune of $65,000,000, and Imerys Talc America Inc. for $2.5 million. Plaintiffs, having sought $285 million, were no doubt disappointed. The Giannecchini verdict was the third large verdict in the Missouri talc litigation. See Myron Levin, “Johnson & Johnson Hammered Again in Talc-Ovarian Cancer Verdict of $70 Million,” (Oct. 27, 2016); Brandon Lowrey, “J & J, Talc Co. Hit With $70M Baby Powder Cancer Verdict,” Law360 (Oct. 2016).

In his closing argument, Giannecchini’s lawyer, R. Allen Smith, reportedly accused Johnson & Johnson of having “rigged” regulatory agencies to ignore the dangers of talc, and of having “falsified” medical records to hide the problem. Smith implored the jury to “make them stop”; make them “stop this madness.”

Make them stop the madness, indeed. The November 2016 issue of Epidemiology features a publication of the “Sister Study,” which explored whether there was any association between perineal talc use and ovarian cancer. The authors acknowledged, as had Judge Johnson in the Carl case, that some prior case-control studies had found an increased risk of ovarian cancer, but that prospective cohort studies have not confirmed an association. Nicole L. Gonzalez, Katie M. O’Brien, Aimee A. D’Aloisio, Dale P. Sandler, and Clarice R. Weinberg, “Douching, Talc Use, and Risk of Ovarian Cancer,” 27 Epidemiology 797 (2016).

The Sister Study (2003–2009) followed a cohort of 50,884 women whose sisters had been diagnosed with breast cancer. Talc use was ascertained at baseline, before diagnosis of subsequent disease and before any chance for selective recall. The cohort was followed for a median of 6.6 years, in which time there were 154 cases of ovarian cancer during the follow up, available for analysis using Cox’s proportional hazards model. Perineal talc use at baseline was not associated with later ovarian cancer. The authors reported a hazard ratio of 0.73, less than expected, with a 95% confidence interval of 0.44, 1.2.

So, yes, make them stop this madness; close the gate.

Posted in Causation, Expert Witnesses, Rule 702, Scientific Evidence | Comments Off on Talc Litigation – Stop the Madness

National Academies’ Teaching Modules on Scientific Policy Issues

June 30th, 2016

Today, the National Academies of Sciences, Engineering, and Medicine announced its release of nine teaching modules to help public policy decision makers and students in professional schools understand the role of science in policy decision making.[1] The modules were developed by university faculty members for the use of other faculty who want to help their students appreciate the complexity and nuances of the evidence for and against scientific claims.

A group within the Academies’ Committee on Science, Technology and the Law supervised the development of the teaching modules, which are now publicly available at the Academies’ website. The Committee was chaired by Paul Brest, former dean and professor emeritus (active), Stanford Law School, and Saul Perlmutter, Franklin W. and Karen Weber Dabby Chair, University of California, Berkeley, and senior scientist, E.O. Lawrence Berkeley National Laboratory. The Gordon and Betty Moore Foundation and the National Biomedical Research Foundation sponsored the development of the modules.

The modules use case studies to illustrate basic scientific and statistical principles involved in contemporary scientific issues that have significant policy implications. The modules are designed to help future policy and decision makers understand and evaluate the scientific evidence that they will doubtlessly encounter. To date, nine modules have been developed and released, in the hope that they will serve as references and examples for future teaching modules.

The nine modules prepared to date are:

Models: Scientific Practice in Context

prepared by:
– Elizabeth Fisher, Professor of Environmental Law, Faculty of Law and Corpus Christi College, Oxford University
– Pasky Pascual, Environmental Protection Agency
– Wendy Wagner, Joe A. Worsham Centennial Professor, University of Texas at Austin School of Law

The Interpretation of DNA Evidence: A Case Study in Probabilities

prepared by:

– David H. Kaye, Associate Dean for Research and Distinguished Professor, The Pennsylvania State University (Penn State Law)

Translating Science into Policy: The Role of Decision Science

prepared by:

– Paul Brest, Former Dean and Professor Emeritus (active), Stanford Law School

Placing a Bet: A New Therapy for Parkinson’s Disease

prepared by:

– Kevin W. Sharer, Senior Lecturer, Harvard Business School, Harvard University

Shale Gas Development

prepared by:

– John D. Graham, Dean, School of Public and Environmental Affairs, Indiana University
– John A. Rupp, Adjunct Instructor, School of Public and Environmental Affairs, and Senior Research Scientist, Indiana Geological Survey, Indiana University
– Adam V. Maltese, Associate Professor of Science Education, School of Education, and Adjunct Faculty in Department of Geological Sciences, Indiana University

Drug-Induced Birth Defects: Exploring the Intersection of Regulation, Medicine, Science, and Law

prepared by:

– Nathan A. Schachtman, Lecturer in Law, Columbia Law School

Vaccines

prepared by:

– Arturo Casadevall, Professor and Chair, W. Harry Feinstone Department of Molecular Microbiology and Immunology, Johns Hopkins University Bloomberg School of Public Health

Forensic Pattern Recognition Evidence

prepared by:

– Simon A. Cole, Professor, Department of Criminology, Law, and Society, Director, Newkirk Center for Science and Society, University of California, Irvine
– Alyse Berthental, Ph.D. Candidate, Department of Criminology, Law, and Society, University of California, Irvine
– Jaclyn Seelagy, Scholar, PULSE (Program on Understanding Law, Science, and Evidence), University of California, Los Angeles School of Law

Scientific Evidence of Factual Causation

prepared by:

– Steve C. Gold, Professor of Law, Rutgers School of Law-Newark
– Michael D. Green, Williams Professor of Law, Wake Forest University School of Law
– Joseph Sanders, A.A. White Professor of Law, University of Houston Law Center

[1] See “Academies Release Educational Modules to Help Future Policymakers and Other Professional-School Students Understand the Role of Science in Decision Making” (June 30, 2016).

Posted in Causation, Scientific Evidence, statistical evidence | Comments Off on National Academies’ Teaching Modules on Scientific Policy Issues

Credible Incredulity

May 19th, 2016

Has skepticism become a victim of political correctness and adversarial zeal?

In the last century, philosopher Bertrand Russell advanced intelligent skepticism against myriad enthusiams and mindless beliefs, political, religious, and pseudo-scientific. Russell saw unwarranted certainty as a serious intellectual offense:

“The fundamental cause of the trouble is that in the modern world the stupid are cocksure while the intelligent are full of doubt.”

Bertrand Russell, “The Triumph of Stupidity” (1933), Mortals and Others: Bertrand Russell’s American Essays, 1931-1935 , at 28 (1998). When many American intellectuals were still in their love swoon over Stalin, Russell chastised the Soviet dictator for his betrayal of ideals and his enslavement of Eastern European. Stalinism’s certainty about politics and science was not a virtue, but a grave sin. Or, in Russell’s words:

“One of the painful things about our time is that those who feel certainty are stupid, and those with any imagination and understanding are filled with doubt and indecision.”

Bertrand Russell, New Hopes for a Changing World at 4-5 (1951).

In the 21st century, ideologues of various stripes have tried to silence healthy skepticism and doubt by claiming that their critics have “manufactured doubt.”[1] This aggression against skepticism and doubt, joined with a biased conception of conflicts of interest, have become part of a concerted campaign to privilege tendentious scientific claims from critical scrutiny.

Philosopher Susan Haack, who has aligned herself on occasion with these politicized acolytes of certainty,[2] recently has pushed back, with a reminder that credulity for unwarranted claims, in all walks of life, is unethical.[3] Haack’s essay is a delightful effort to clarify what credulity is, and to explore why credulity is an epistemologic vice and a social hazard, as well as the implications for citizens and scientists of living in an evidence-based, not a faith-based world.

Drawing inspiration from the the English mathematician and philosopher, William Kingdon Clifford, Haack has adopted one of Clifford’s bon mots as her motto:

“The credulous man is father to the liar and the cheat.”[4]

Indeed! And credulous judges and juries are the parents to specious claims and shyster lawyers.

Clifford’s essay should be required reading for politicians, judges, regulators, and legislators who evaluate the claims of scientist advocates. Spurning ethical relativism, Clifford identified the key intellectual “sin” in an evidence-based world:

“It is wrong always, everywhere, and for anyone, to believe anything upon insufficient evidence.”

William K. Clifford, “The Ethics of Belief,” 29 Contemporary Rev. 289, 295 (1877).

Professor Haack should be commended for her fulsome irony for publishing in a journal of one of the world’s more credulous institutions, and for reminding us that credulity is an intellectual vice.

[1] See, e.g., David Michaels, Doubt is Their Product: How Industry’s Assault on Science Threatens Your Health (2008); Naomi Oreskes and Erik M. Conway, Merchants of Doubt: How a Handful of Scientists Obscured the Truth on Issues from Tobacco Smoke to Global Warming (2010).

[2] See, e.g., “Bendectin, Diclegis & The Philosophy of Science” (Oct. 26, 2013).

[3] Susan Haack, “Credulity and Circumspection: Epistemological Character and the Ethics of Belief,” 88 Proc. Am. Catholic Philosophical Ass’n 27 (2015).

[4] citing and quoting William K. Clifford, “The Ethics of Belief ” (1877), in Leslie Stephen and Sir Frederick Pollock, eds., The Ethics of Belief and Other Essays 70, 77 (London 1947).

Posted in Scientific Evidence, Scientific Misconduct | Comments Off on Credible Incredulity

The IARC Process is Broken

May 4th, 2016

Last spring, the International Agency for Research on Cancer (IARC) convened a working group that voted to classify the herbicide glyphosate as “probably carcinogenic to humans.” The vote was followed by IARC’s Press Release, a summary in The Lancet,[1] and the publication of a “monograph,” volume 112 in the IARC series.

IARC classifications of a chemical as “probably” carcinogenic to humans are actually fairly meaningless exercises in semantics, not science. A close reading of the IARC Preamble definition of probable reveals that probable does not mean greater than 50%:

“The terms probably carcinogenic and possibly carcinogenic have no quantitative significance and are used simply as descriptors of different levels of evidence of human carcinogenicity, with probably carcinogenic signifying a higher level of evidence than possibly carcinogenic.”

Despite the vacuity of the IARC’s “probability” determinations, IARC decisions have serious real-world consequences in the realm of regulation and litigation. Monsanto, the manufacturer of glyphosate herbicide, reacted strongly, expressing “outrage” and claiming that the IARC had cherry picked data to reach its conclusion. Jack Kaskey, “Monsanto ‘Outraged’ by Assessment That Roundup Probably Causes Cancer,” 43 Product Safety & Liability Reporter 416 (Mar. 30, 2015).

In the wake of the IARC classification, in the fall of 2015, the United States Environmental Protection Agency (EPA) reviewed the evidence for, and against, glysophate’s carcinogenicity. The EPA found that the IARC had deliberately failed to consider studies that did not find associations, and that the complete scientific record did not support a conclusion of human carcinogenicity. EPA Report of the Cancer Assessment Review Committee on Glyphosate (Oct. 1, 2015).

For undisclosed reasons, however, the EPA’s report was never made public until a couple of weeks ago, when it showed up briefly on the agency’s website, only to be pulled down after a day or so. See David Schultz, “EPA Panel Finds Glyphosate Not Likely to Cause Cancer,” Product Safety & Liability Reporter (May 03, 2016). No doubt the present Administration viewed a conflict between EPA and IARC, and disparaging comments about the IARC’s “process” to be national security issues. At the very least, the Administration would not want to undermine the litigation industry’s reliance upon the IARC cherry-picked report.

All joking aside, the incident highlights the problematic nature of the IARC decision process, and the reliance of regulatory agencies on the apparent authority of IARC determinations. The IARC process is toxic and should be remediated.

[1] Kathryn Z Guyton, Dana Loomis, Yann Grosse, Fatiha El Ghissassi, Lamia Benbrahim-Tallaa, Neela Guha, Chiara Scoccianti, Heidi Mattock, Kurt Straif, on behalf of the International Agency for Research on Cancer Monograph Working Group, IARC, Lyon, France, “Carcinogenicity of tetrachlorvinphos, parathion, malathion, diazinon, and glyphosate,” 16 The Lancet Oncology 490 (2015).

Posted in Causation, Scientific Evidence, Scientific Misconduct | Comments Off on The IARC Process is Broken

The Education of Judge Rufe – The Zoloft MDL

April 9th, 2016

The Honorable Cynthia M. Rufe is a judge on the United States District Court, for the Eastern District of Pennsylvania. Judge Rufe was elected to a judgeship on the Bucks County Court of Common Pleas in 1994. She was appointed to the federal district court in 2002. Like most state and federal judges, little in her training and experience as a lawyer prepared her to serve as a gatekeeper of complex expert witness scientific opinion testimony. And yet, the statutory code of evidence, and in particular, Federal Rules of Evidence 702 and 703, requires her do just that.

The normal approach to MDL cases is marked by the Field of Dreams: “if you build it, they will come.” Last week, Judge Rufe did something that is unusual in pharmaceutical litigation; she closed the gate and sent everyone home. In re Zoloft Prod. Liab. Litig., MDL NO. 2342, 12-MD-2342, 2016 WL 1320799 (E.D. Pa. April 5, 2016).

Her Honor’s decision was hardly made in haste. The MDL began in 2012, and proceeded in a typical fashion with case management orders that required the exchange of general causation expert witness reports. The plaintiffs’ steering committee (PSC), acting for the plaintiffs, served the report of only one epidemiologist, Anick Bérard, who took the position that Zoloft causes virtually every major human congenital anomaly known to medicine. The defendants challenged the admissibility of Bérard’s opinions. After extensive briefings and evidentiary hearings, the trial court found that Bérard’s opinions were riddled with inconsistent assessments of studies, eschewed generally accepted methods of causal inference, ignored contrary evidence, adopted novel, unreliable methods of endorsing “trends” in studies, and failed to address epidemiologic studies that did not support her subjective opinions. In re Zoloft Prods. Liab. Litig., 26 F. Supp. 3d 449 (E.D.Pa.2014). The trial court permitted plaintiffs an opportunity to seek reconsideration of Bérard’s exclusion, which led to the trial court’s reaffirming its previous ruling. In re Zoloft Prods. Liab. Litig., No. 12–md–2342, 2015 WL 314149, at *2 (E.D.Pa. Jan. 23, 2015).

Notwithstanding the PSC’s claims that Bérard was the best qualified expert witness in her field and that she was the only epidemiologist needed to support the plaintiffs’ causal claims, the MDL court indulged the PSC by permitting plaintiffs another bite at the apple. Over defendants’ objections, the court permitted the PSC to name yet another expert witness, statistician Nicholas Jewell, to do what Bérard had failed to do: proffer an opinion on general causation supported by sound science. In re Zoloft Prods. Liab. Litig., No. 12–md–2342, 2015 WL 115486, at * 2 (E.D.Pa. Jan. 7, 2015).

As a result of this ruling, the MDL dragged on for over a year, in which time, the PSC served a report by Jewell, and then the defendants conducted a discovery deposition of Jewell, and lodged a new Rule 702 challenge. Although Jewell brought more statistical sophistication to the task, he could not transmute lead into gold; nor could he support the plaintiffs’ causal claims without committing most of the same fallacies found in Bérard’s opinions. After another round of Rule 702 briefs and hearings, the MDL court excluded Jewell’s unwarranted causal opinions. In re Zoloft Prods. Liab. Litig., No. 12–md–2342, 2015 WL 7776911 (E.D.Pa. Dec. 2, 2015).

The successive exclusions of Bérard and Jewell left the MDL court in a peculiar position. There were other witnesses, Robert Cabrera, a teratologist, Michael Levin, a molecular biologist, and Thomas Sadler, an embryologist, whose opinions addressed animal toxicologic studies, biological plausibility, and putative mechanisms. These other witnesses, however, had little or no competence in epidemiology, and they explicitly relied upon Bérard’s opinions with respect to human outcomes. As a result of Bérard’s exclusion, these witnesses were left free to offer their views about what happens in animals at high doses, or about theoretical mechanisms, but they were unable to address human causation.

Although the PSC had no expert witnesses who could legitimately offer reasonably supported opinions about the causation of human birth defects, the plaintiffs refused to decamp and leave the MDL forum. Faced with the prospect of not trying their cases to juries, the PSC instead tried the patience of the MDL judge. The PSC pulled out the stops in adducing weak, irrelevant, and invalid evidence to support their claims, sans epidemiologic expertise. The PSC argued that adverse event reports, internal company documents that discussed possible associations, the biological plausibility opinions of Levin and Sadler, the putative mechanism opinions of Cabrera, differential diagnoses offered to support specific causation, and the hip-shot opinions of a former-FDA-commissioner-for-hire, David Kessler could come together magically to supply sufficient evidence to have their cases submitted to juries. Judge Rufe saw through the transparent effort to manufacture evidence of causation, and granted summary judgment on all remaining Zoloft cases in the MDL. s In re Zoloft Prod. Liab. Litig., MDL NO. 2342, 12-MD-2342, 2016 WL 1320799, at *4 (E.D. Pa. April 5, 2016).

After a full briefing and hearing on Bérard’s opinion, a reconsideration of Bérard, a permitted “do over” of general causation with Jewell, a full briefing and hearing on Jewell’s opinions, the MDL court was able to deal deftly with the snippets of evidence “cobbled together” to substitute for evidence that might support a conclusion of causation. The PSC’s cobbled case was puffed up to give the appearance of voluminous evidence, in 200 exhibits that filled six banker’s boxes. Id. at *5. The ruse was easily undone; most of the exhibits and purported evidence were obvious rubbish. “The quantity of the evidence is not, however, coterminous with the quality of evidence with regard to the issues now before the Court.” Id. The banker’s boxes contained artifices such as untranslated foreign-language documents, and company documents relating to the development and marketing of the medication. The PSC resubmitted reports from Levin, Cabrera, and Sadler, whose opinions were already adjudicated to be incompetent, invalid, irrelevant, or inadequate to support general causation. The PSC pointed to the specific causation opinions of a clinical cardiologist, Ra-Id Abdulla, M.D., who proffered dubious differential etiologies, ruling in Zoloft as a cause of individual children’s birth defects, despite his inability to rule out truly known and unknown causes in the differential reasoning. The MDL court, however, recognized that “[a] differential diagnosis assumes that general causation has been established,” id. at *7, and that Abdulla could not bootstrap general causation by purporting to reach a specific causation opinion (even if those specific causation opinions were legitimate).

The PSC submitted the recent consensus statement of the American Statistical Association (ASA)[1], which it misrepresented to be an epidemiologic study. Id. at *5. The consensus statement makes some pedestrian pronouncements about the difference between statistical and clinical significance, about the need for other considerations in addition to statistical significance, in supporting causal claims, and the lack of bright-line distinctions for statistical significance in assessing causality. All true, but immaterial to the PSC’s expert witnesses’ opinions that over-endorsed statistical significance in the few instances in which it was shown, and over-interpreted study data that was based upon data mining and multiple comparisons, in blatant violation of the ASA’s declared principles.

Stretching even further for “human evidence,” the PSC submitted documentary evidence of adverse event reports, as though they could support a causal conclusion.[2] There are about four million live births each year, with an expected rate of serious cardiac malformations of about one per cent.[3] The prevalence of SSRI anti-depressant use is at least two per cent, which means that we would expect 800 cardiac birth defects each year to occur in children of mother’s who took SSRI anti-depressants in the first trimester. If Zoloft had an average market share of all the SSRIs of about 25 per cent, then 200 cardiac defects each year would occur in children born to mothers who took Zoloft. Given that Zoloft has been on the market since the early 1990s, we would expect that there would be thousands of children, exposed to Zoloft during embryogenesis, born with cardiac defects, if there was nothing untoward about maternal exposure to the medication. Add the stimulated reporting of adverse events from lawyers, lawyer advertising, and lawyer instigation, you have manufactured evidence not probative of causation at all.[4] The MDL court cut deftly and swiftly through the smoke screen:

“These reports are certainly relevant to the generation of study hypotheses, but are insufficient to create a material question of fact on general causation.”

Id. at *9. The MDL court recognized that epidemiology was very important in discerning a causal connection between a common exposure and a common outcome, especially when the outcome has an expected rate in the general population. The MDL court stopped short of holding that epidemiologic evidence was required (which on the facts of the case would have been amply justified), but instead supported its ratio decidendi on the need to account for the extant epidemiology that contradicted or failed to support the strident and subjective opinions of the plaintiffs’ expert witnesses. The MDL court thus gave plaintiffs every benefit of the doubt by limiting its holding on the need for epidemiology to:

“when epidemiological studies are equivocal or inconsistent with a causation opinion, experts asserting causation opinions must thoroughly analyze the strengths and weaknesses of the epidemiological research and explain why that body of research does not contradict or undermine their opinion.”

Id. at *5, quoting from In re Zoloft Prods. Liab. Litig., 26 F. Supp. 3d 449, 476 (E.D. Pa. 2014).

The MDL court also saw through the thin veneer of respectability of the testimony of David Kessler, a former FDA commissioner who helped make large fortunes for some of the members of the PSC by the feeding frenzy he created with his moratorium on silicone gel breast implants. Even viewing Kessler’s proffered testimony in the most charitable light, the court recognized that he offered little support for a causal conclusion other than to delegate the key issues to epidemiologists. Id. at *9. As for the boxes of regulatory documents, foreign labels, and internal company memoranda, the MDL court found that these documents did not raise a genuine issue of material fact concerning general causation:

“Neither these documents, nor draft product documents or foreign product labels containing language that advises use of birth control by a woman taking Zoloft constitute an admission of causation, as opposed to acknowledging a possible association.”

Id.

In the end, the MDL court found that the PSC’s many banker boxes of paper contained too much of nothing for the issue at hand. Having put the defendants through the time and expense of litigating and re-litigating these issues, nothing short of dismissing the pending cases was a fair and appropriate outcome to the Zoloft MDL.

_______________________________________

Given the denouement of the Zoloft MDL, it is worth considering the MDL judge’s handling of the scientific issues raised, misrepresented, argued, or relied upon by the parties. Judge Rufe was required, by Rules 702 and 703, to roll up her sleeves and assess the methodological validity of the challenged expert witnesses’ opinions. That Her Honor was able to do this is a testament to her hard work. Zoloft was not Judge Rufe’s first MDL, and she clearly learned a lot from her previous judicial assignment to an MDL for Avandia personal injury actions.

On May 21, 2007, the New England Journal of Medicine published online a seriously flawed meta-analysis of cardiovascular disease outcomes and rosiglitazone (Avandia) use. See Steven E. Nissen, M.D., and Kathy Wolski, M.P.H., “Effect of Rosiglitazone on the Risk of Myocardial Infarction and Death from Cardiovascular Causes,” 356 New Engl. J. Med. 2457 (2007). The Nissen article did not appear in print until June 14, 2007, but the first lawsuits resulted within a day or two of the in-press version. The lawsuits soon thereafter reached a critical mass, with the inevitable creation of a federal court Multi-District Litigation.

Within a few weeks of Nissen’s article, the Annals of Internal Medicine published an editorial by Cynthia Mulrow, and other editors, in which questioned the Nissen meta-analysis[5], and introduced an article that attempted to replicate Nissen’s work[6]. The attempted replication showed that the only way Nissen could have obtained his nominally statistically significant result was to have selected a method, Peto’s fixed effect method, known to be biased for use with clinical trials with uneven arms. Random effect methods, more appropriate for the clinically heterogeneous clinical trials, consistently failed to replicate the Nissen result. Other statisticians weighed in and pointed out that using the risk difference made much more sense when there were multiple trials with zero events in one or the other or both arms of the trials. Trials with zero cardiovascular events in both arms represented important evidence of low, but equal risk, of heart attacks, which should be captured in an appropriate analysis. When the risk difference approach was used, with exact statistical methods, there was no statistically significant increase in risk in the dataset used by Nissen.[7] Other scientists, including some of Nissen’s own colleagues at the Cleveland Clinic, and John Ioannidis, weighed in to note how fragile and insubstantial the Nissen meta-analysis was[8]:

“As rosiglitazone case demonstrates, minor modifications of the meta-analysis protocol can change the statistical significance of the result. For small effects, even the direction of the treatment effect estimate may change.”

Nissen achieved his political objective with his shaky meta-analysis. The FDA convened an Advisory Committee meeting, which in turn resulted in a negative review of the safety data, and the FDA’s imposition of warnings and a Risk Evaluation and Mitigation Strategy, which all but prohibited use of rosiglizone.[9] A clinical trial, RECORD, had already started, with support from the drug sponsor, GlaxoSmithKline, which fortunately was allowed to continue.

On a parallel track to the regulatory activities, the federal MDL, headed by Judge Rufe, proceeded to motions and a hearing on GSK’s Rule 702 challenge to plaintiffs’ evidence of general causation. The federal MDL trial judge denied GSK’s motions to exclude plaintiffs’ causation witnesses in an opinion that showed significant diffidence in addressing scientific issues. In re Avandia Marketing, Sales Practices and Product Liability Litigation, 2011 WL 13576, *12 (E.D. Pa. 2011). See “Learning to Embrace Flawed Evidence – The Avandia MDL’s Daubert Opinion” (Jan. 10, 2011.

After Judge Rufe denied GSK’s challenges to the admissibility of plaintiffs’ expert witnesses’ causation opinions in the Avandia MDL, the RECORD trial was successfully completed and published.[10] RECORD was a long term, prospectively designed randomized cardiovascular trial in over 4,400 patients, followed on average of 5.5 yrs. The trial was designed with a non-inferiority end point of ruling out a 20% increased risk when compared with standard-of-care diabetes treatment The trial achieved its end point, with a hazard ratio of 0.99 (95% confidence interval, 0.85-1.16) for cardiovascular hospitalization and death. A readjudication of outcomes by the Duke Clinical Research Institute confirmed the published results.

On Nov. 25, 2013, after convening another Advisory Committee meeting, the FDA announced the removal of most of its restrictions on Avandia:

“Results from [RECORD] showed no elevated risk of heart attack or death in patients being treated with Avandia when compared to standard-of-care diabetes drugs. These data do not confirm the signal of increased risk of heart attacks that was found in a meta-analysis of clinical trials first reported in 2007.”

FDA Press Release, “FDA requires removal of certain restrictions on the diabetes drug Avandia” (Nov. 25, 2013). And in December 2015, the FDA abandoned its requirement of a Risk Evaluation and Mitigation Strategy for Avandia. FDA, “Rosiglitazone-containing Diabetes Medicines: Drug Safety Communication – FDA Eliminates the Risk Evaluation and Mitigation Strategy (REMS)” (Dec. 16, 2015).

GSK’s vindication came too late to reverse Judge Rufe’s decision in the Avandia MDL. GSK spent over six billion dollars on resolving Avandia claims. And to add to the company’s chagrin, GSK lost patent protection for Avandia in April 2012.[11]

Something good, however, may have emerged from the Avandia litigation debacle. Judge Rufe heard from plaintiffs’ expert witnesses in Avandia about the hierarchy of evidence, about how observational studies must be evaluated for bias and confounding, about the importance of statistical significance, and about how studies that lack power to find relevant associations may still yield conclusions with appropriate meta-analysis. Important nuances of meta-analysis methodology may have gotten lost in the kerfuffle, but given that plaintiffs had reasonable quality clinical trial data, Avandia plaintiffs’ counsel could eschew their typical reliance upon weak and irrelevant lines of evidence, based upon case reports, adverse event disproportional reporting, and the like.

The Zoloft litigation introduced Judge Rufe to a more typical pharmaceutical litigation. Because the outcomes of interest were birth defects, there were no clinical trials. To be sure, there were observational epidemiologic studies, but now the defense expert witnesses were carefully evaluating the studies for bias and confounding, and the plaintiffs’ expert witnesses were double counting studies and ignoring multiple comparisons and validity concerns. Once again, in the Zoloft MDL, plaintiffs’ expert witnesses made their non-specific complaints about “lack of power” (without ever specifying the relevant alternative hypothesis), but it was the defense expert witnesses who cited relevant meta-analyses that attempted to do something about the supposed lack of power. Plaintiffs’ expert witnesses inconsistently argued “lack of power” to disregard studies that had outcomes that undermined their opinions, even when those studies had narrow confidence intervals surrounding values at or near 1.0.

The Avandia litigation laid the foundation for Judge Rufe’s critical scrutiny by exemplifying the nature and quantum of evidence to support a reasonable scientific conclusion. Notwithstanding the mistakes made in the Avandia litigation, this earlier MDL created an invidious distinction with the Zoloft PSC’s evidence and arguments, which looked as weak and insubstantial as they really were.

[1] Ronald L. Wasserstein & Nicole A. Lazar, “The ASA’s Statement on p-Values: Context, Process, and Purpose,” The American Statistician, available online (Mar. 7, 2016), in-press at DOI:10.1080/00031305.2016.1154108, <http://dx.doi.org/10.1080/>. See “The American Statistical Association’s Statement on and of Significance” (Mar. 17, 2016); “The ASA’s Statement on Statistical Significance – Buzzing from the Huckabees” (Mar. 19, 2016).

[2] See 21 C.F.R. § 314.80 (a) Postmarketing reporting of adverse drug experiences (defining “[a]dverse drug experience” as “[a]ny adverse event associated with the use of a drug in humans, whether or not considered drug related”).

[3] See Centers for Disease Control and Prevention, “Birth Defects Home Page” (last visited April 8, 2016).

[4] See, e.g., Derrick J. Stobaugh, Parakkal Deepak, & Eli D. Ehrenpreis, “Alleged isotretinoin-associated inflammatory bowel disease: Disproportionate reporting by attorneys to the Food and Drug Administration Adverse Event Reporting System,” 69 J. Am. Acad. Dermatol. 393 (2013) (documenting stimulated reporting from litigation activities).

[5] Cynthia D. Mulrow, John Cornell & A. Russell Localio, “Rosiglitazone: A Thunderstorm from Scarce and Fragile Data,” 147 Ann. Intern. Med. 585 (2007).

[6] George A. Diamond, Leon Bax & Sanjay Kaul, “Uncertain Effects of Rosiglitazone on the Risk for Myocardial Infartion and Cardiovascular Death,” 147 Ann. Intern. Med. 578 (2007).

[7] Tian, et al., “Exact and efficient inference procedure for meta-analysis and its application to the analysis of independent 2 × 2 tables with all available data but without artificial continuity correction” 10 Biostatistics 275 (2008)

[8] Adrian V. Hernandez, Esteban Walker, John P.A. Ioannidis, and Michael W. Kattan, “Challenges in meta-analysis of randomized clinical trials for rare harmful cardiovascular events: the case of rosiglitazone,” 156 Am. Heart J. 23, 28 (2008).

[9] Janet Woodcock, FDA Decision Memorandum (Sept. 22, 2010).

[10] Philip D. Home, et al., “Rosiglitazone evaluated for cardiovascular outcomes in oral agent combination therapy for type 2 diabetes (RECORD): a multicentre, randomised, open-label trial,” 373 Lancet 2125 (2009).

[11] “Pharmacovigilantism – Avandia Litigation” (Nov. 27, 2013).

Posted in Causation, Expert Witnesses, Meta-analysis, Rule 702, Scientific Evidence, statistical evidence | Comments Off on The Education of Judge Rufe – The Zoloft MDL

Expert Witness – Ghost Busters

March 29th, 2016

Andrew Funkhouser was tried and convicted for selling cocaine. On appeal, the Missouri Court of Appeals affirmed his conviction and his sentence of prison for 30 years. State v. Funkhouser, 729 S.W.2d 43 (Mo. App. 1987). On a petition for post-conviction relief, Funkhouser asserted that he was deprived of his Sixth Amendment right to effective counsel. Funkhouser v. State, 779 S.W.2d 30 (Mo. App. 1989).

One of the alleged grounds of ineffectiveness was his lawyer’s failure to object to the prosecutor’s cross-examination of a defense expert witness, clinical psychologist Frederick Nolen, on Nolan’s belief in ghosts. Id. at 32. On direct examination, Nolen testified that he had published or presented on multiple personalities, hypnosis, and ghosts.

On cross-examination, the prosecution inquired of Nolan about his theory of ghosts:

“Q. Doctor, I believe that you’ve done some work in the theory of ghosts, is that right?

A. Yes.

Q. I believe you told me that some of that work you’d based on your own experiences, is that correct?

A. Yes.

Q. You also told me you have lived in a haunted house for 13 years, is that right?

A. Yes.

Q. You have seen the ghost, is that correct?

A. Yes.”

Id. at 32-33. Funkhouser asserted that the cross-examination was improper because his expert witness was examined on his religious beliefs, and his counsel was ineffective for failing to object. Id. at 33. The Missouri Court of Appeals disagreed. Counsel are permitted to cross-examine an adversary’s expert witness

“in any reasonable respect that will test his qualifications, credibility, skill or knowledge and the value and accuracy of his opinions.”

The court held that any failure to object could not be incompetence because the examination was proper. Id.

So there you have it: wacky beliefs systems are fair game for cross-examination of expert witnesses, at least in the “Show-Me” state.

And this broad scope of cross-examination is probably a good thing because almost anything seems to go in Missouri. The Show-Me state has been wiping up the rear in the law of expert witness admissibility. Missouri Revised Statutes contains a version of the Federal Rule of Evidence 702, which goes back to the language before the federal statutory revision in 2000:

Expert witness, opinion testimony admissible–hypothetical question not required, when.

490.065. 1. In any civil action, if scientific, technical or other specialized knowledge will assist the trier of fact to understand the evidence or to determine a fact in issue, a witness qualified as an expert by knowledge, skill, experience, training, or education may testify thereto in the form of an opinion or otherwise.

In January 2016, the Missouri state senate passed a bill that would bring the Missouri standard in line with the current federal court rule of evidence. Most of the Republican senators voted for the bill; none of the Democrats voted in favor of the reform. Chris Semones, “Missouri: One Step Closer to Daubert,” in Expert Witness Network (Jan. 26, 2016).

Posted in Expert Witnesses, Rule 702, Scientific Evidence | Comments Off on Expert Witness – Ghost Busters

The ASA’s Statement on Statistical Significance – Buzzing from the Huckabees

March 19th, 2016

People say crazy things. In a radio interview, Evangelical Michael Huckabee argued that the Kentucky civil clerk who refused to issue a marriage license to a same-sex couple was as justified in defying an unjust court decision as people are justified in disregarding Dred Scott v. Sanford, 60 U.S. 393 (1857), which Huckabee described as still the “law of the land.”¹ Chief Justice Roger B. Taney would be proud of Huckabee’s use of faux history, precedent, and legal process to argue his cause. Definition of “huckabee”: a bogus factoid.

Consider the case of Sander Greenland, who attempted to settle a score with an adversary’s expert witness, who had opined in 2002, that Bayesian analyses were rarely used at the FDA for reviewing new drug applications. The adversary’s expert witness obviously got Greenland’s knickers in a knot because Greenland wrote an article in a law review of all places, in which he presented his attempt to “correct the record” and show how the statement of the opposing expert witness was“ludicrous” .^²To support his indictment on charges of ludicrousness, Greenland ignored the FDA’s actual behavior in reviewing new drug applications,^³ and looked at the practice of the Journal of Clinical Oncology, a clinical journal published 24 issues a year, with occasional supplements. Greenland found the word “Bayesian” 50 times in over 40,000 journal pages, and declared victory. According to Greenland, “several” (unquantified) articles had used Bayesian methods to explore, post hoc, statistically nonsignificant results.”^⁴

Given Greenland’s own evidence, the posterior odds that Greenland was correct in his charges seem to be disturbingly low, but he might have looked at the published papers that conducted more serious, careful surveys of the issue.^⁵ This week, the Journal of the American Medical Association published yet another study by John Ioannidis and colleagues, which documented actual practice in the biomedical literature. And no surprise, Bayesian methods barely register in a systematic survey of the last 25 years of published studies. See David Chavalarias, Joshua David Wallach, Alvin Ho Ting Li, John P. A. Ioannidis, “Evolution of reporting P values in the biomedical literature, 1990-2015,” 315 J. Am. Med. Ass’n 1141 (2016). See also Demetrios N. Kyriacou, “The Enduring Evolution of the P Value,” 315 J. Am. Med. Ass’n 1113 (2016) (“Bayesian methods are not frequently used in most biomedical research analyses.”).

So what are we to make of Greenland’s animadversions in a law review article? It was a huckabee moment.

Recently, the American Statistical Association (ASA) issued a statement on the use of statistical significance and p-values. In general, the statement was quite moderate, and declined to move in the radical directions urged by some statisticians who attended the ASA’s meeting on the subject. Despite the ASA’s moderation, the ASA’s statement has been met with huckabee-like nonsense and hyperbole. One author, a pharmacologist trained at the University of Washington, with post-doctoral training at the University of California, Berkeley, and an editor of PloS Biology, was moved to write:

“However, the ASA notes, the importance of the p-value has been greatly overstated and the scientific community has become over-reliant on this one – flawed – measure.”

Lauren Richardson, “Is the p-value pointless?” (Mar. 16, 2016). And yet, no where in the ASA’s statement does the group suggest that the the p-value was a “flawed” measure. Richardson suffered a lapse and wrote a huckabee.

Not surprisingly, lawyers attempting to spin the ASA’s statement have unleashed entire hives of huckabees in an attempt to deflate the methodological points made by the ASA. Here is one example of a litigation-industry lawyer who argues that the American Statistical Association Statement shows the irrelevance of statistical significance for judicial gatekeeping of expert witnesses:

“To put it into the language of Daubert, debates over ‘p-values’ might be useful when talking about the weight of an expert’s conclusions, but they say nothing about an expert’s methodology.”

Max Kennerly, “Statistical Significance Has No Place In A Daubert Analysis” (Mar. 13, 2016) [cited as Kennerly]

But wait; the expert witness must be able to rule out chance, bias and confounding when evaluating a putative association for causality. As Austin Bradford Hill explained, even before assessing a putative association for causality, scientists need first to have observations that

“reveal an association between two variables, perfectly clear-cut and beyond what we would care to attribute to the play of chance.”

Austin Bradford Hill, “The Environment and Disease: Association or Causation?” 58 Proc. Royal Soc’y Med. 295, 295 (1965) (emphasis added).

The analysis of random error is an essential step on the methodological process. Simply because a proper methodology requires consideration of non-statistical factors does not remove the statistical from the methodology. Ruling out chance as a likely explanation is a crucial first step in the methodology for reaching a causal conclusion when there is an “expected value” or base rate of for the outcome of interest in the population being sampled.

Kennerly shakes his hive of huckabees:

“The erroneous belief in an ‘importance of statistical significance’ is exactly what the American Statistical Association was trying to get rid of when they said, ‘The widespread use of “statistical significance” (generally interpreted as p ≤ 0.05)’ as a license for making a claim of a scientific finding (or implied truth) leads to considerable distortion of the scientific process.”

And yet, the ASA never urged that scientists “get rid of” statistical analyses and assessments of attained levels of significance probability. To be sure, they cautioned against overinterpreting p-values, especially in the context of multiple comparisons, non-prespecified outcomes, and the like. The ASA criticized bright-line rules, which are often used by litigation-industry expert witnesses to over-endorse the results of studies with p-values less than 5%, often in the face of multiple comparisons, cherry-picked outcomes, and poorly and incompletely described methods and results. What the ASA described as a “considerable distortion of the scientific process” was claiming scientific truth on the basis of “p < 0.05.” As Bradford Hill pointed out in 1965, a clear-cut association, beyond that which we would care to attribute to chance, is the beginning of the analysis of an association for causality, not the end of it. Kennerly ignores who is claiming “truth” in the litigation context. Defense expert witnesses frequently are opining no more than “not proven.” The litigation industry expert witnesses must opine that there is causation, or else they are out of a job.

The ASA explained that the distortion of the scientific process comes from making a claim of a scientific conclusion of causality or its absence, when the appropriate claim is “we don’t know.” The ASA did not say, suggest, or imply that a claim of causality can be made in the absence of finding statistical significance, and as well as validation of the statistical model on which it is based, and other factors as well. The ASA certainly did not say that the scientific process will be served well by reaching conclusions of causation without statistical significance. What is clear is that statistical significance should not be an abridgment for a much more expansive process. Reviewing the annals of the International Agency for Research on Cancer (even in its currently politicized state), or the Institute of Medicine, an honest observer would be hard pressed to come up with examples of associations for outcomes that have known base rates, which associations were determined to be causal in the absence of studies that exhibited statistical significance, along with many other indicia of causality.

Some other choice huckabees from Kennerly:

“It’s time for courts to start seeing the phrase ‘statistically significant’ in a brief the same way they see words like ‘very,’ ‘clearly,’ and ‘plainly’. It’s an opinion that suggests the speaker has strong feelings about a subject. It’s not a scientific principle.”

Of course, this ignores the central limit theorems, the importance of random sampling, the pre-specification of hypotheses and level of Type I error, and the like. Stuff and nonsense.

And then in a similar vein, from Kennerly:

“The problem is that many courts have been led astray by defendants who claim that ‘statistical significance’ is a threshold that scientific evidence must pass before it can be admitted into court.”

In my experience, litigation-industry lawyers oversell statistical significance rather than defense counsel who may question reliance upon studies that lack it. Kennerly’s statement is not even wrong, however, because defense counsel knowledgeable of the rules of evidence would know that statistical studies themselves are rarely admitted into evidence. What is admitted, or not, is the opinion of expert witnesses, who offer opinions about whether associations are causal, or not causal, or inconclusive.

1 Ben Mathis-Lilley, “Huckabee Claims Black People Aren’t Technically Citizens During Critique of Unjust Laws,” The Slatest (Sept. 11 2015) (“[T]he Dred Scott decision of 1857 still remains to this day the law of the land, which says that black people aren’t fully human… .”).

2 Sander Greenland, “The Need for Critical Appraisal of Expert Witnesses in Epidemiology and Statistics,” 39 Wake Forest Law Rev. 291, 306 (2004). See “The Infrequency of Bayesian Analyses in Non-Forensic Court Decisions” (Feb. 16, 2014).

3 To be sure, eight years after Greenland published this diatribe, the agency promulgated a guidance that set recommended practices for Bayesian analyses in medical device trials. FDA Guidance for the Use of Bayesian Statistics in Medical Device Clinical Trials (February 5, 2010); 75 Fed. Reg. 6209 (February 8, 2010); see also Laura A. Thompson, “Bayesian Methods for Making Inferences about Rare Diseases in Pediatric Populations” (2010); Greg Campbell, “Bayesian Statistics at the FDA: The Trailblazing Experience with Medical Devices” (Presentation give by Director, Division of Biostatistics Center for Devices and Radiological Health at Rutgers Biostatistics Day, April 3, 2009). Even today, Bayesian analysis remains uncommon at the U.S. FDA.

4 39 Wake Forest Law Rev. at 306-07 & n.61 (citing only one paper, Lisa Licitra et al., Primary Chemotherapy in Resectable Oral Cavity Squamous Cell Cancer: A Randomized Controlled Trial, 21 J. Clin. Oncol. 327 (2003)).

5 See, e.g., J. Martin Bland & Douglas G. Altman, “Bayesians and frequentists,” 317 Brit. Med. J. 1151, 1151 (1998) (“almost all the statistical analyses which appear in the British Medical Journal are frequentist”); David S. Moore, “Bayes for Beginners? Some Reasons to Hesitate,” 51 The Am. Statistician 254, 254 (“Bayesian methods are relatively rarely used in practice”); J.D. Emerson & Graham Colditz, “Use of statistical analysis in the New England Journal of Medicine,” in John Bailar & Frederick Mosteler, eds., Medical Uses of Statistics 45 (1992) (surveying 115 original research studies for statistical methods used; no instances of Bayesian approaches counted); Douglas Altman, “Statistics in Medical Journals: Developments in the 1980s,” 10 Statistics in Medicine 1897 (1991); B.S. Everitt, “Statistics in Psychiatry,” 2 Statistical Science 107 (1987) (finding only one use of Bayesian methods in 441 papers with statistical methodology).

Posted in Causation, Rule 702, Scientific Evidence, statistical evidence | Comments Off on The ASA’s Statement on Statistical Significance – Buzzing from the Huckabees

The American Statistical Association’s Statement on and of Significance

March 17th, 2016

In scientific circles, some commentators have so zealously criticized the use of p-values that they have left uninformed observers with the impression that random error was not an interesting or important consideration in evaluating the results of a scientific study. In legal circles, counsel for the litigation industry and their expert witnesses have argued duplicitously that statistical significance was at once both unimportant, except when statistical significance is observed, in which causation is conclusive. The recently published Statement of the American Statistical Association (“ASA”) restores some sanity to the scientific and legal discussions of statistical significance and p-values. Ronald L. Wasserstein & Nicole A. Lazar, “The ASA’s Statement on p-Values: Context, Process, and Purpose,” The American Statistician, available online (Mar. 7, 2016), in-press at DOI:10.1080/00031305.2016.1154108, <http://dx.doi.org/10.1080/>.

Recognizing that sound statistical practice and communication affects research and public policy decisions, the ASA has published a statement of interpretative principles for statistical significance and p-values. The ASA’s statement first, and foremost, points out that the soundness of scientific conclusions turns on more than statistical methods alone. Study design, conduct, and evaluation often involve more than a statistical test result. And the ASA goes on to note, contrary to the contrarians, that “the p-value can be a useful statistical measure,” although this measure of attained significance probability “is commonly misused and misinterpreted.” ASA at 7. No news there.

The ASA’s statement puts forth six principles, all of which have substantial implications for how statistical evidence is received and interpreted in courtrooms. All are worthy of consideration by legal actors – legislatures, regulators, courts, lawyers, and juries.

“1. P-values can indicate how incompatible the data are with a specified statistical model.”

The ASA notes that a p-value shows the “incompatibility between a particular set of data and a proposed model for the data.” Although there are some in the statistical world who rail against null hypotheses of no association, the ASA reports that “[t]he most common context” for p-values consists of a statistical model that includes a set of assumptions, including a “null hypothesis,” which often postulates the absence of association between exposure and outcome under study. The ASA statement explains:

“The smaller the p-value, the greater the statistical incompatibility of the data with the null hypothesis, if the underlying assumptions used to calculate the p-value hold. This incompatibility can be interpreted as casting doubt on or providing evidence against the null hypothesis or the underlying assumptions.”

Some lawyers want to overemphasize statistical significance when present, but to minimize the importance of statistical significance when it is absent. They will find no support in the ASA’s statement.

“2. P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.”

Of course, there are those who would misinterpret the meaning of p-values, but the flaw lies in the interpreters, not in the statistical concept.

“3. Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold.”

Note that the ASA did not say that statistical significance is irrelevant to scientific conclusions. Of course, statistical significance is but one factor, which does not begin to account for study validity, data integrity, or model accuracy. The ASA similarly criticizes the use of statistical significance as a “bright line” mode of inference, without consideration of the contextual considerations of “the design of a study, the quality of the measurements, the external evidence for the phenomenon under study, and the validity of assumptions that underlie the data analysis.” Criticizing the use of “statistical significance” as singularly assuring the correctness of scientific judgment does not, however, mean that “statistical significance” is irrelevant or unimportant as a consideration in a much more complex decision process.

“4. Proper inference requires full reporting and transparency”

The ASA explains that the proper inference from a p-value can be completely undermined by “multiple analyses” of study data, with selective reporting of sample statistics that have attractively low p-values, or cherry picking of suggestive study findings. The ASA points out that common practices of selective reporting compromises valid interpretation. Hence the correlative recommendation:

“Researchers should disclose the number of hypotheses explored during the study, all data collection decisions, all statistical analyses conducted and all p-values computed. Valid scientific conclusions based on p-values and related statistics cannot be drawn without at least knowing how many and which analyses were conducted, and how those analyses (including p-values) were selected for reporting.”

ASA Statement. See also “Courts Can and Must Acknowledge Multiple Comparisons in Statistical Analyses” (Oct. 14, 2014).

“5. A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.”

The ASA notes the commonplace distinction between statistical and practical significance. The independence between statistical and practice significance does not, however, make statistical significance irrelevant, especially in legal and regulatory contexts, in which parties claim that a risk, however small, is relevant. Of course, we want the claimed magnitude of association to be relevant, but we also need the measured association to be accurate and precise.

“6. By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis.”

Of course, a p-value cannot validate the model, which is assumed to generate the p-value. Contrary to the hyperbolic claims one sees in litigation, the ASA notes that “a p-value near 0.05 taken by itself offers only weak evidence against the null hypothesis.” And so the ASA counsels that “data analysis should not end with the calculation of a p-value when other approaches are appropriate and feasible.”

What is important, however, is that the ASA never suggests that significance testing or measurement of significance probability is not an important and relevant part of the process. To be sure, the ASA notes that because of “the prevalent misuses of and misconceptions concerning p-values, some statisticians prefer to supplement or even replace p-values with other approaches.”

First of these other methods unsurprisingly is estimation with assessment of confidence intervals, although the ASA also includes Bayesian and other methods as well. There are some who express irrational exuberance about the protential of Bayesian methods to restore confidence in scientific process and conclusions. Bayesian approaches are less manipulated than frequentist ones, largely because very few people use Bayesian methods, and even fewer people really understand them.

In some ways, Bayesian statistical approaches are like Apple computers. The Mac OS is less vulnerable to viruses, compared with Windows, because its lower market share makes it less attractive to virus code writers. As Apple’s OS has gained market share, its vulnerability has increased. (My Linux computer on the other hand is truly less vulnerable to viruses because of system architecture, but also because Linux personal computers have almost no market share.) If Bayesian methods become more prevalent, my prediction is that they will be subject to as much abuse as frequent views. The ASA wisely recognized that the “reproducibility crisis” and loss of confidence in scientific research were mostly due to bias, both systematic and cognitive, in how studies are done, interpreted, and evaluated.

Posted in Rule 702, Scientific Evidence, statistical evidence | Comments Off on The American Statistical Association’s Statement on and of Significance