TORTINI

For your delectation and delight, desultory dicta on the law of delicts.

Archives »

Earthquake-Induced Data Loss – We’re All Shook Up

June 26th, 2015

Adam Marcus and Ivan Oransky are medical journalists who publish the Retraction Watch blog. Their blog’s coverage of error, fraud, plagiarism, and other publishing disasters is often first-rate, and a valuable curative for the belief that peer review publication, as it is now practiced, ensures trustworthiness.

Yesterday, Retraction Watch posted an article on earthquake-induced data loss. Shannon Palus, “Lost your data? Blame an earthquake” (June 25, 2015). A commenter on PubPeer raised concerns about a key figure in a paper[1]. The authors acknowledged a problem, which they traced to their loss of data in an earthquake. The journal retracted the paper.

This is not the first instance of earthquake-induced loss of data.

When John O’Quinn and his colleagues in the litigation industry created the pseudo-science of silicone-induced autoimmunity, they recruited Nir Kossovsky, a pathologist at UCLA Medical Center. Although Kossovsky looked a bit like Pee-Wee Herman, he was a graduate of the University of Chicago Pritzker School of Medicine, and the U.S. Naval War College, and a consultant to the FDA. In his dress whites, Kossovsky helped O’Quinn sell his silicone immunogenicity theories to juries and judges around the country. For a while, the theories sold well.

In testifying and dodging discovery for the underlying data in his silicone studies, Kossovsky was as slick as silicone itself. Ultimately, when defense counsel subpoenaed the underlying data from Kossovsky’s silicone study, Kossovsky shrugged and replied that the Northridge Earthquake destroyed his data. Apparently coffee cups and other containers of questionable fluids spilled on his silicone data in the quake, and Kossovsky’s emergency response was to obtain garbage cans and throw out the data. For the gory details, see Gary Taubes, “Silicone in the System: Has Nir Kossovsky really shown anything about the dangers of breast implants?” Discover Magazine (Dec. 1995).

As Mr. Taubes points out, Kossovsky’s paper was rejected by several journals before being published in the Journal of Applied Biomaterials, of which Kossovsky was a member of the editorial board. The lack of data did not, however, keep Kossovsky from continuing to testify, and from trying to commercialize, along with his wife, Beth Brandegee, and his father, Ram Kossowsky[2], an ELISA-based silicone “antibody” biomarker diagnostic test, Detecsil. Although Rule 702 had been energized by the Daubert decision in 1993, many judges were still not willing to take a hard look at Kossovsky’s study, his test, or to demand the supposedly supporting data. The Food and Drug Administration, however, eventually caught up with Kossovsky, and the Detecsil marketing ceased. Lillian J. Gill, FDA Acting Director, Office of Compliance, Letter to Beth S. Brandegee, President, Structured Biologicals (SBI) Laboratories: Detecsil Silicone Sensitivity Test (July 15, 1994); see Taubes, Discover Magazine.

After defense counsel learned of the FDA’s enforcement action against Kossovsky and his company, the litigation industry lost interest in Kossovsky, and his name dropped off trial witness lists. His name also dropped off the rolls of tenured UCLA faculty, and he apparently left medicine altogether to become a business consultant. Dr. Kossovsky became “an authority on business process risk and reputational value.” Kossovsky is now the CEO and Director of Steel City Re, which specializes in strategies for maintaining and enhancing reputational value. Ironic; eh?

A review of PubMed’s entries for Nir Kossovsky shows that his run in silicone started in 1983, and ended in 1996. He testified for plaintiffs in Hopkins v. Dow Corning Corp., 33 F.3d 1116 (9th Cir.1994) (tried in 1991), and in the infamous case of Johnson v. Bristol-Myers Squibb, CN 91-21770, Tx Dist. Ct., 125th Jud. Dist., Harris Cty., 1992.

A bibliography of Kossovsky silicone oeuvre is listed, below.

[1] Federico S. Rodríguez, Katterine A. Salazar, Nery A. Jara, María A García-Robles, Fernando Pérez, Luciano E. Ferrada, Fernando Martínez, and Francisco J. Nualart, “Superoxide-dependent uptake of vitamin C in human glioma cells,” 127 J. Neurochemistry 793 (2013).

[2] Father and son apparently did not agree on how to spell their last name.

Nir Kossovsky, D. Conway, Ram Kossowsky & D. Petrovich, “Novel anti-silicone surface-associated antigen antibodies (anti-SSAA(x)) may help differentiate symptomatic patients with silicone breast implants from patients with classical rheumatological disease,” 210 Curr. Topics Microbiol. Immunol. 327 (1996)

Nir Kossovsky, et al., “Preservation of surface-dependent properties of viral antigens following immobilization on particulate ceramic delivery vehicles,” 29 J. Biomed. Mater. Res. 561 (1995)

E.A. Mena, Nir Kossovsky, C. Chu, and C. Hu, “Inflammatory intermediates produced by tissues encasing silicone breast prostheses,” 8 J. Invest. Surg. 31 (1995)

Nir Kossovsky, “Can the silicone controversy be resolved with rational certainty?” 7 J. Biomater. Sci. Polymer Ed. 97 (1995)

Nir Kossovsky & C.J. Freiman, “Physicochemical and immunological basis of silicone pathophysiology,” 7 J. Biomater. Sci. Polym. Ed. 101 (1995)

Nir Kossovsky, et al., “Self-reported signs and symptoms in breast implant patients with novel antibodies to silicone surface associated antigens [anti-SSAA(x)],” 6 J. Appl. Biomater. 153 (1995), and “Erratum,” 6 J. Appl. Biomater. 305 (1995)

Nir Kossovsky & J. Stassi, “A pathophysiological examination of the biophysics and bioreactivity of silicone breast implants,” 24s1 Seminars Arthritis & Rheum. 18 (1994)

Nir Kossovsky & C.J. Freiman, “Silicone breast implant pathology. Clinical data and immunologic consequences,” 118 Arch. Pathol. Lab. Med. 686 (1994)

Nir Kossovsky & C.J. Freiman, “Immunology of silicone breast implants,” 8 J. Biomaterials Appl. 237 (1994)

Nir Kossovsky & N. Papasian, “Mammary implants,” 3 J. Appl. Biomater. 239 (1992)

Nir Kossovsky, P. Cole, D.A. Zackson, “Giant cell myocarditis associated with silicone: An unusual case of biomaterials pathology discovered at autopsy using X-ray energy spectroscopic techniques,” 93 Am. J. Clin. Pathol. 148 (1990)

Nir Kossovsky & R.B. Snow RB, “Clinical-pathological analysis of failed central nervous system fluid shunts,” 23 J. Biomed. Mater. Res. 73 (1989)

R.B. Snow & Nir Kossovsky, “Hypersensitivity reaction associated with sterile ventriculoperitoneal shunt malfunction,” 31 Surg. Neurol. 209 (1989)

Nir Kossovsky & Ram Kossowsky, “Medical devices and biomaterials pathology: Primary data for health care technology assessment,” 4 Internat’l J. Technol. Assess. Health Care 319 (1988)

Nir Kossovsky, John P. Heggers, and M.C. Robson, “Experimental demonstration of the immunogenicity of silicone-protein complexes,” 21 J. Biomed. Mater. Res. 1125 (1987)

Nir Kossovsky, John P. Heggers, R.W. Parsons, and M.C. Robson, “Acceleration of capsule formation around silicone implants by infection in a guinea pig model,” 73 Plastic & Reconstr. Surg. 91 (1984)

John Heggers, Nir Kossovsky, et al., “Biocompatibility of silicone implants,” 11 Ann. Plastic Surg. 38 (1983)

Nir Kossovsky, John P. Heggers, et al., “Analysis of the surface morphology of recovered silicone mammary prostheses,” 71 Plast. Reconstr. Surg. 795 (1983)

Posted in Expert Witness Hall of Shame, Expert Witnesses, Rule 702, Scientific Evidence, Scientific Misconduct | Comments Off on Earthquake-Induced Data Loss – We’re All Shook Up

Diclegis and Vacuous Philosophy of Science

June 24th, 2015

Just when you thought that nothing more could be written intelligently about the Bendectin litigation, you find out you are right. Years ago, Michael Green and Joseph Sanders each wrote thoughtful, full-length books[1] about the litigation assault on the morning-sickness (nausea and vomiting of pregnancy) medication, which was voluntarily withdrawn by its manufacturer from the United States market. Dozens, if not hundreds, of law review articles discuss the scientific issues, the legal tactics, and the judicial decisions in the U.S. Bendectin litigation, including the Daubert case in Supreme Court and in the Ninth Circuit. But perhaps fresh eyes might find something new and fresh to say.

Boaz Miller teaches social epistemology and philosophy of science, and he recently weighed in on the role that scientific consensus played in resolving the Bendectin litigation. Miller, “Scientific Consensus and Expert Testimony in Courts: Lessons from the Bendectin Litigation,” Foundations of Science (2014) (Oct. 17, 2014) (in press) [cited as Miller]. Miller astutely points out that scientific consensus may or may not be epistemic, that is, based upon robust, valid, sufficient scientific evidence of causality. Scientists are people, and sometimes they come to conclusions based upon invalid evidence, or because of cognitive biases, or social pressures, and the like. Sometimes scientists get the right result for the wrong reasons. From this position he argues that adverting to scientific consensus is fraught with danger of being misled, and that the Bendectin ligitation specifically is an example of courts led astray by a “non-epistemic” scientific consensus. Miller at 1.

Miller is correct that the scientific consensus on Bendectin’s safety, which emerged after the initiation of litigation, played a role in resolving the litigation, id. at 8, but he badly misunderstands how the consensus actually operated to bring closure to the birth defects litigation. Remarkably, he pays no attention to the consolidated trial of over 800 cases before the Hon. Carl B. Rubin, in the Southern District of Ohio. This trial resulted in a defense verdict in March 1985, and judgment that withstood appellate review. Assoc. Press, “U.S. Jury Clears a Nausea Drug in Birth Defects,” N.Y. Times (Mar. 13, 1985). The subsequent litigation turned into guerilla warfare based upon relatively few remaining individual cases in state and federal courts. In one of the state court cases, the trial court appointed neutral expert witnesses, who opined that plaintiffs had failed to make out their causal claims of teratogenicity in human beings. DePyper v. Navarro, No. 83–303467-NM, 1995 WL 788828 (Mich. Cir. Ct. Nov. 27, 1995).

To be sure, plaintiffs’ expert witnesses and plaintiffs’ counsel continued in their campaign to manufacture “reasonable medical certainty” of Bendectin’s teratogenicity, well after a scientific consensus emerged. Boaz Miller makes the stunning claim that this consensus was not a “knowledge-based” consensus because:

(1) the research was controlled by parties to the dispute (Miller at 10);

(2) the consensus ignored or diminished the “value” of in vitro toxicology (Miller at 13);

(3) the consensus relied upon most heavily upon the epidemiologic evidence (Miller at 14);

(4) the animal toxicology research was “prematurely” abandoned when the U.S. withdrew its product from the market (Miller at 15); and

(5) the withdrawal ended the “threat” to public health, and the concerns about teratogenicity (Miller at 15).

Miller’s asserted reasons are demonstrably incorrect. Although Merrell Richardson funded some studies early on, by the time the scientific consensus emerged, many studies funded by neutral sources, and conducted by researchers of respected integrity, were widely available. The consensus did not diminish the value of in vivo toxicology; rather scientists evaluated the available evidence through their understanding of epidemiology’s superiority in assessing actual risks in human populations. Animal studies were not prematurely abandoned; more accurately, the animal studies gave way to more revealing, valid studies in humans about human outcomes. The sponsor’s withdrawal of Bendectin in the United States was not the cause of any abandonment of research. The drug remained available outside the United States, in countries with less rapacious tort systems, and researchers would have, in any event, continued animal studies if there were something left open by previous research. A casual browse through PubMed’s collection of articles on thalidomide shows that animal research continued well after that medication had been universally withdrawn for use in pregnant women. Given that thalidomide was unquestionably a human teratogen, there was a continued interest in understanding its teratogenicity. No such continued interest existed for Bendectin after the avalanche of exculpatory human data.

What sort of inquiry permitted Miller to reach his conclusions? His article cites no studies, no whole-animal toxicology, no in vitro research, no epidemiologic studies, no systematic reviews, no regulatory agency reviews, and no meta-analysis. All exist in abundance. The full extent of his engagement with the actual scientific data and issues is a reference to an editorial and two letters to the editor[2]! From the exchange of views in one set of letters in 1985, Miller infers that there was “clear dissent within the community of toxicologists.” Miller at 13. The letters in question, however, were written in a journal of teratology, which was not limited to toxicology, and the interlocutors were well aware of the hierarchy of evidence that placed human observational studies at the top of the evidential pyramid.

Miller argues that it was possible that the consensus was not knowledge-based because it might have reflected the dominance of epidemiology over the discipline of toxicology. Again, he ignores the known dubious validity of inferring human teratogenicity from high dose whole animal or in vitro toxicology. By the time the scientific consensus emerged with respect to Bendectin’s safety, this validity point was widely appreciated by all but the most hardened rat killers, and plaintiffs’ counsel. In less litigious countries, the drug never left the market. No regulatory agency ever called for its withdrawal.

Miller might have tested whether the scientific community’s consensus on Bendectin, circa 1992 (when Daubert was being briefed in the Supreme Court) was robust by looking to how well it stood up to further testing. He did not, but he could easily have found the following. The U.S. sponsor of Diclegis, Duchesnay USA, sought and obtained the indication for its medication in pregnancy. Under U.S. law, Duchesnay’s new drug application had to establish safety and efficacy for this indication. In 2013, the U.S. FDA approved Bendectin, under the tradename, Diclegis[3], as a combination of doxylamine succinate and pyridoxine hydrochloride for sale in the United States. Under the FDA’s pregnancy labeling system, Diclegis is a category A, with a specific indication for use in pregnancy. The FDA’s review of the actual data is largely available for all to see. See, e.g., Center for Drug Evaluation and Research, Other Reviews (Aug. 2012); Summary Review (April 2013); Pharmacology Review (March 2013); Medical Review (March 2013); Statistical Review (March 2013); Cross Discipline Team Leader Review (April 2013). Given the current scientific record, the consensus that emerged in the early 1990s looks strong. Indeed, the consensus was epistemically strong when reached two decades ago.

Miller is certainly correct that reliance upon consensus entails epistemic risks. Sometimes the consensus has not looked very hard or critically at all the evidence. Political, financial, and cognitive biases can be prevalent. Miller fails to show that any such biases were prevalent in the early 1990s, or that they infected judicial assessments of the plaintiffs’ causal claims in Bendectin litigation. Miller is also wrong to suggest that courts did not look beyond the consensus to the actual evidential base for plaintiffs’ claims. Through the lens of expert witness testimony, both party and court-appointed expert witnesses, courts and juries had a better view of the causation issues than Miller appreciates. Miller’s philosophy of science might be improved by his rolling up his sleeves and actually looking at the data[4].

[1] See Joseph Sanders, Bendectin on Trial: A Study of Mass Tort Litigation (1998); Michael D. Green, Bendectin and Birth Defects: The Challenges of Mass Toxic Substances Litigation (1996).

[2] Robert Brent, “Editorial comment on comments on ‘Teratogen Update: Bendectin’,” 31 Teratology 429 (1985); Kenneth S. Brown, John M. Desesso, John Hassell, Norman W. Klein, Jon M. Rowland, A. J. Steffek, Betsy D. Carlton, Cas. Grabowski, William Slikker Jr. and David Walsh, “Comments on ‘Teratogen Update: Bendectin’,” 31Teratology 431 (1985); Lewis B. Holmes, “Response to comments on ‘Teratogen Update: Bendectin’,” 31 Teratology 432 (1985).

[3] See FDA News Release, “FDA approves Diclegis for pregnant women experiencing nausea and vomiting,” (April 8, 2013). The return of this drug to the United States market was held up as a triumph of science over the will of the industry litigation. See Gideon Koren, “The Return to the USA of the Doxylamine-Pyridoxine Delayed Release Combination (Diclegis®) for Morning Sickness — A New Morning for American Women,” 20 J. Popul. Ther. Clin. Pharmacol. e161 (2013).

[4] See “Bendectin, Diclegis & The Philosophy of Science” (Oct 26, 2013).

Posted in Rule 702, Scientific Evidence | Comments Off on Diclegis and Vacuous Philosophy of Science

How to Fake a Moon Landing

June 8th, 2015

“The meaning of the world is the separation of wish and fact.”
Kurt Gödel

Everyone loves science except when science defeats wishes for a world not known. Coming to accept the world based upon evidence requires separating wish from fact. And when the evidence is lacking in quality or quantity, then science requires us to have the discipline to live with uncertainty rather than wallow in potentially incorrect beliefs.

Darryl Cunningham has written an engaging comic graphics book about science and the scientific worldview. Darryl Cunningham, How to Fake a Moon Landing: Exposing the Myths of Science Denial (2013). Through pictorial vignettes, taken from current controversies, Cunningham has created a delightful introduction to scientific methodology and thinking. Cunningham has provided chapters on several modern scandalous deviations from the evidence-based understanding of the world, including:

  • The Moon Hoax
  • Homeopathy
  • Chiropractic
  • The MMR Vaccination Scandal
  • Evolution
  • Fracking
  • Climate Change, and
  • Science Denial

Most people will love this book. Lawyers will love the easy-to-understand captions. Physicians will love the debunking of chiropractic. Republicans will love the book’s poking fun at (former Dr.) Andrew Wakefield and his contrived MMR vaccination-autism scare, and the liberal media’s unthinking complicity in his fraud. Democrats will love the unraveling of the glib, evasive assertions of the fracking industry. New Agers will love the book because of its neat pictures, and they probably won’t read the words anyway, and so they will likely miss the wonderful deconstruction of homeopathy and other fashionable hokum. Religious people, however, will probably hate the fun poked at all attempts to replace evidence with superstitions.Roblox HackBigo Live Beans HackYUGIOH DUEL LINKS HACKPokemon Duel HackRoblox HackPixel Gun 3d HackGrowtopia HackClash Royale Hackmy cafe recipes stories hackMobile Legends HackMobile Strike Hack

Without rancor, Cunningham pillories all true believers who think that they can wish the facts of the world. At $16.95, the book is therapeutic and a bargain.

Posted in Scientific Evidence | Comments Off on How to Fake a Moon Landing

Judicial Control of the Rate of Error in Expert Witness Testimony

May 28th, 2015

In Daubert, the Supreme Court set out several criteria or factors for evaluating the “reliability” of expert witness opinion testimony. The third factor in the Court’s enumeration was whether the trial court had considered “the known or potential rate of error” in assessing the scientific reliability of the proffered expert witness’s opinion. Daubert v. Merrell Dow Pharmaceuticals, Inc., 509 U.S. 579, 593 (1993). The Court, speaking through Justice Blackmun, failed to provide much guidance on the nature of the errors subject to gatekeeping, on how to quantify the errors, and on to know how much error was too much. Rather than provide a taxonomy of error, the Court lumped “accuracy, validity, and reliability” together with a grand pronouncement that these measures were distinguished by no more than a “hen’s kick.” Id. at 590 n.9 (1993) (citing and quoting James E. Starrs, “Frye v. United States Restructured and Revitalized: A Proposal to Amend Federal Evidence Rule 702,” 26 Jurimetrics J. 249, 256 (1986)).

The Supreme Court’s failure to elucidate its “rate of error” factor has caused a great deal of mischief in the lower courts. In practice, trial courts have rejected engineering opinions on stated grounds of their lacking an error rate as a way of noting that the opinions were bereft of experimental and empirical evidential support[1]. For polygraph evidence, courts have used the error rate factor to obscure their policy prejudices against polygraphs, and to exclude test data even when the error rate is known, and rather low compared to what passes for expert witness opinion testimony in many other fields[2]. In the context of forensic evidence, the courts have rebuffed objections to random-match probabilities that would require that such probabilities be modified by the probability of laboratory or other error[3].

When it comes to epidemiologic and other studies that require statistical analyses, lawyers on both sides of the “v” frequently misunderstand p-values or confidence intervals to provide complete measures of error, and ignore the larger errors that result from bias, confounding, study validity (internal and external), inappropriate data synthesis, and the like[4]. Not surprisingly, parties fallaciously argue that the Daubert criterion of “rate of error” is satisfied by expert witness’s reliance upon studies that in turn use conventional 95% confidence intervals and measures of statistical significance in p-values below 0.05[5].

The lawyers who embrace confidence intervals and p-values as their sole measure of error rate fail to recognize that confidence intervals and p-values are means of assessing only one kind of error: random sampling error. Given the carelessness of the Supreme Court’s use of technical terms in Daubert, and its failure to engage in the actual evidence at issue in the case, it is difficult to know whether the Court intended to suggest that random error was the error rate it had in mind[6]. The statistics chapter in the Reference Manual on Scientific Evidence helpfully points out that the inferences that can be drawn from data turn on p-values and confidence intervals, as well as on study design, data quality, and the presence or absence of systematic errors, such as bias or confounding.  Reference Manual on Scientific Evidence at 240 (3d 2011) [Manual]. Random errors are reflected in the size of p-values or the width of confidence intervals, but these measures of random sampling error ignore systematic errors such as confounding and study biases. Id. at 249 & n.96.

The Manual’s chapter on epidemiology takes an even stronger stance: the p-value for a given study does not provide a rate of error or even a probability of error for an epidemiologic study:

“Epidemiology, however, unlike some other methodologies—fingerprint identification, for example—does not permit an assessment of its accuracy by testing with a known reference standard. A p-value provides information only about the plausibility of random error given the study result, but the true relationship between agent and outcome remains unknown. Moreover, a p-value provides no information about whether other sources of error – bias and confounding – exist and, if so, their magnitude. In short, for epidemiology, there is no way to determine a rate of error.”

Manual at 575. This stance seems not entirely justified given that there are Bayesian approaches that would produce credibility intervals accounting for sampling and systematic biases. To be sure, such approaches have their own problems and they have received little to no attention in courtroom proceedings to date.

The authors of the Manual’s epidemiology chapter, who are usually forgiving of judicial error in interpreting epidemiologic studies, point to one United States Court of Appeals case that fallaciously interpreted confidence intervals magically to quantify bias and confounding in a Bendectin birth defects case. Id. at 575 n. 96[7]. The Manual could have gone further to point out that, in the context of multiple studies, of different designs and analyses, cognitive biases involved in evaluating, assessing, and synthesizing the studies are also ignored by statistical measures such as p-values and confidence intervals. Although the Manual notes that assessing the role of chance in producing a particular set of sample data is “often viewed as essential when making inferences from data,” the Manual never suggests that random sampling error is the only kind of error that must be assessed when interpreting data. The Daubert criterion would appear to encompass all varieties or error, not just random error.

The Manual’s suggestion that epidemiology does not permit an assessment of the accuracy of epidemiologic findings misrepresents the capabilities of modern epidemiologic methods. Courts can, and do, invoke gatekeeping approaches to weed out confounded study findings. See “Sorting Out Confounded Research – Required by Rule 702” (June 10, 2012). The “reverse Cornfield inequality” was an important analysis that helped establish the causal connection between tobacco smoke and lung cancer[8]. Olav Axelson studied and quantified the role of smoking as a confounder in epidemiologic analyses of other putative lung carcinogens.[9] Quantitative methods for identifying confounders have been widely deployed[10].

A recent study in birth defects epidemiology demonstrates the power of sibling cohorts in addressing the problem of residual confounding from observational population studies with limited information about confounding variables. Researchers looking at various birth defect outcomes among offspring of women who used certain antidepressants in early pregnancy generally found no associations in pooled data from Iceland, Norway, Sweden, Finland, and Denmark. A putative association between maternal antidepressant use and a specific kind of cardiac defect (right ventricular outflow tract obstruction or RVOTO) did appear in the overall analysis, but was reversed when the analysis was limited to the sibling subcohort. The study found an apparent association between RVOTO defects and first trimester maternal exposure to selective serotonin reuptake inhibitors, with an adjusted odds ratio of 1.48 (95% C.I., 1.15, 1.89). In the adjusted analysis for siblings, the study found an OR of 0.56 (95% C.I., 0.21, 1.49) in an adjusted sibling analysis[11]. This study and many others show how creative analyses can elucidate and quantify the direction and magnitude of confounding effects in observational epidemiology.

Systematic bias has also begun to succumb to more quantitative approaches. A recent guidance paper by well-known authors encourages the use of quantitative bias analysis to provide estimates of uncertainty due to systematic errors[12].

Although the courts have failed to articulate the nature and consequences of erroneous inference, some authors would reduce all of Rule 702 (and perhaps 704, 403 as well) to a requirement that proffered expert witnesses “account” for the known and potential errors in their opinions:

“If an expert can account for the measurement error, the random error, and the systematic error in his evidence, then he ought to be permitted to testify. On the other hand, if he should fail to account for any one or more of these three types of error, then his testimony ought not be admitted.”

Mark Haug & Emily Baird, “Finding the Error in Daubert,” 62 Hastings L.J. 737, 739 (2011).

Like most antic proposals to revise Rule 702, this reform vision shuts out the full range of Rule 702’s remedial scope. Scientists certainly try to identify potential sources of error, but they are not necessarily very good at it. See Richard Horton, “Offline: What is medicine’s 5 sigma?” 385 Lancet 1380 (2015) (“much of the scientific literature, perhaps half, may simply be untrue”). And as Holmes pointed out[13], certitude is not certainty, and expert witnesses are not likely to be good judges of their own inferential errors[14]. Courts continue to say and do wildly inconsistent things in the course of gatekeeping. Compare In re Zoloft (Setraline Hydrochloride) Products, 26 F. Supp. 3d 449, 452 (E.D. Pa. 2014) (excluding expert witness) (“The experts must use good grounds to reach their conclusions, but not necessarily the best grounds or unflawed methods.”), with Gutierrez v. Johnson & Johnson, 2006 WL 3246605, at *2 (D.N.J. November 6, 2006) (denying motions to exclude expert witnesses) (“The Daubert inquiry was designed to shield the fact finder from flawed evidence.”).

[1] See, e.g., Rabozzi v. Bombardier, Inc., No. 5:03-CV-1397 (NAM/DEP), 2007 U.S. Dist. LEXIS 21724, at *7, *8, *20 (N.D.N.Y. Mar. 27, 2007) (excluding testimony from civil engineer about boat design, in part because witness failed to provide rate of error); Sorto-Romero v. Delta Int’l Mach. Corp., No. 05-CV-5172 (SJF) (AKT), 2007 U.S. Dist. LEXIS 71588, at *22–23 (E.D.N.Y. Sept. 24, 2007) (excluding engineering opinion that defective wood-carving tool caused injury because of lack of error rate); Phillips v. Raymond Corp., 364 F. Supp. 2d 730, 732–33 (N.D. Ill. 2005) (excluding biomechanics expert witness who had not reliably tested his claims in a way to produce an accurate rate of error); Roane v. Greenwich Swim Comm., 330 F. Supp. 2d 306, 309, 319 (S.D.N.Y. 2004) (excluding mechanical engineer, in part because witness failed to provide rate of error); Nook v. Long Island R.R., 190 F. Supp. 2d 639, 641–42 (S.D.N.Y. 2002) (excluding industrial hygienist’s opinion in part because witness was unable to provide a known rate of error).

[2] See, e.g., United States v. Microtek Int’l Dev. Sys. Div., Inc., No. 99-298-KI, 2000 U.S. Dist. LEXIS 2771, at *2, *10–13, *15 (D. Or. Mar. 10, 2000) (excluding polygraph data based upon showing that claimed error rate came from highly controlled situations, and that “real world” situations led to much higher error (10%) false positive error rates); Meyers v. Arcudi, 947 F. Supp. 581 (D. Conn. 1996) (excluding polygraph in civil action).

[3] See, e.g., United States v. Ewell, 252 F. Supp. 2d 104, 113–14 (D.N.J. 2003) (rejecting defendant’s objection to government’s failure to quantify laboratory error rate); United States v. Shea, 957 F. Supp. 331, 334–45 (D.N.H. 1997) (rejecting objection to government witness’s providing separate match and error probability rates).

[4] For a typical judicial misstatement, see In re Zoloft Products, 26 F. Supp.3d 449, 454 (E.D. Pa. 2014) (“A 95% confidence interval means that there is a 95% chance that the ‘‘true’’ ratio value falls within the confidence interval range.”).

[5] From my experience, this fallacious argument is advanced by both plaintiffs’ and defendants’ counsel and expert witnesses. See also Mark Haug & Emily Baird, “Finding the Error in Daubert,” 62 Hastings L.J. 737, 751 & n.72 (2011).

[6] See David L. Faigman, et al. eds., Modern Scientific Evidence: The Law and Science of Expert Testimony § 6:36, at 359 (2007–08) (“it is easy to mistake the p-value for the probability that there is no difference”)

[7] Brock v. Merrell Dow Pharmaceuticals, Inc., 874 F.2d 307, 311-12 (5th Cir. 1989), modified, 884 F.2d 166 (5th Cir. 1989), cert. denied, 494 U.S. 1046 (1990). As with any error of this sort, there is always the question whether the judges were entrapped by the parties or their expert witnesses, or whether the judges came up with the fallacy on their own.

[8] See Joel B Greenhouse, “Commentary: Cornfield, Epidemiology and Causality,” 38 Internat’l J. Epidem. 1199 (2009).

[9] Olav Axelson & Kyle Steenland, “Indirect methods of assessing the effects of tobacco use in occupational studies,” 13 Am. J. Indus. Med. 105 (1988); Olav Axelson, “Confounding from smoking in occupational epidemiology,” 46 Brit. J. Indus. Med. 505 (1989); Olav Axelson, “Aspects on confounding in occupational health epidemiology,” 4 Scand. J. Work Envt’l Health 85 (1978).

[10] See, e.g., David Kriebel, Ariana Zeka1, Ellen A Eisen, and David H. Wegman, “Quantitative evaluation of the effects of uncontrolled confounding by alcohol and tobacco in occupational cancer studies,” 33 Internat’l J. Epidem. 1040 (2004).

[11] Kari Furu, Helle Kieler, Bengt Haglund, Anders Engeland, Randi Selmer, Olof Stephansson, Unnur Anna Valdimarsdottir, Helga Zoega, Miia Artama, Mika Gissler, Heli Malm, and Mette Nørgaard, “Selective serotonin reuptake inhibitors and ventafaxine in early pregnancy and risk of birth defects: population based cohort study and sibling design,” 350 Brit. Med. J. 1798 (2015).

[12] Timothy L.. Lash, Matthew P. Fox, Richard F. MacLehose, George Maldonado, Lawrence C. McCandless, and Sander Greenland, “Good practices for quantitative bias analysis,” 43 Internat’l J. Epidem. 1969 (2014).

[13] Oliver Wendell Holmes, Jr., Collected Legal Papers at 311 (1920) (“Certitude is not the test of certainty. We have been cock-sure of many things that were not so.”).

[14] See, e.g., Daniel Kahneman & Amos Tversky, “Judgment under Uncertainty:  Heuristics and Biases,” 185 Science 1124 (1974).

Posted in Cognitive Biases, Expert Witnesses, Reference Manual on Scientific Evidence, Rule 702, Scientific Evidence | Comments Off on Judicial Control of the Rate of Error in Expert Witness Testimony

Professor Bernstein’s Critique of Regulatory Daubert

May 15th, 2015

In the law of expert witness gatekeeping, the distinction between scientific claims made in support of litigation positions and claims made in support of regulations is fundamental. In re Agent Orange Product Liab. Litig., 597 F. Supp. 740, 781 (E.D.N.Y. 1984) (“The distinction between avoidance of risk through regulation and compensation for injuries after the fact is a fundamental one”), aff’d 818 F.2d 145 (2d Cir. 1987), cert. denied sub nom. Pinkney v. Dow Chemical Co., 487 U.S. 1234 (1988). Although scientists proffer opinions in both litigation and regulatory proceedings, their opinions are usually evaluated by substantially different standards. In federal litigation, civil and criminal, expert witnesses must be qualified and have an epistemic basis for their opinions, to satisfy the statutory requirements of Federal Rule of Evidence 702, and they must have reasonably relied upon otherwise inadmissible evidence (such as the multiple layers of hearsay involved in an epidemiologic study) under Rule 703. In regulatory proceedings, scientists are not subject to admissibility requirements and the sufficiency requirements set by the Administrative Procedures Act are extremely low[1].

Some industry stakeholders are aggrieved by the low standards for scientific decision making in certain federal agencies, and they have urged that the more stringent litigation evidentiary rules be imported into regulatory proceedings. There are several potential problems with such reform proposals. First, the epistemic requirements of science generally, or of Rules 702 and 703 in particular, are not particularly stringent. Scientific method leads to plenty of false positive and false negative conclusions, which are subject to daily challenge and revision. Scientific inference is not necessarily so strict, as much as ordinary reasoning is so flawed, inexact, and careless. Second, the call for “regulatory Daubert” ignores mandates of some federal agency enabling statutes and guiding regulations, which call for precautionary judgments, and which allow agencies to decide issues on evidentiary display that fall short of epistemic warrants for claims of knowledge.

Many lawyers who represent industry stakeholders have pressed for extension of Daubert-type gatekeeping to federal agency decision making. The arguments for constraining agency action find support in the over-extended claims that agencies and so-called public interest science advocates make in support of agency measures. Advocates and agency personnel seem to believe that worst-case scenarios and overstated safety claims are required as “bargaining” positions to achieve the most restrictive and possibly the most protective regulation that can be gotten from the administrative procedure, while trumping industry’s concerns about costs and feasibility. Still, extending Daubert to regulatory proceedings could have the untoward result of lowering the epistemic bar for both regulators and litigation fact finders.

In a recent article, Professor David Bernstein questions the expansion of Daubert into some regulatory realms. David E. Bernstein, “What to Do About Federal Agency Science: Some Doubts About Regulatory Daubert,” 22 Geo. Mason L. Rev. 549 (2015)[cited as Bernstein]. His arguments are an important counterweight to those who insist on changing agency rulemaking and actions at every turn. As an acolyte and a defender of scientific scruples and reasoning in the courts, Bernstein’s arguments are worth taking seriously.

Bernstein reminds us that bad policy, as seen in regulatory agency rulemaking or decisions, is not always a scientific issue. In any event, regulatory actions, unlike jury decisions, are not, or at least should not be, “black boxes.” The agency’s rationale and reasoning are publicly stated, subject to criticism, and open to revision. Jury decisions are opaque, non-transparent, potentially unreasoned, not carefully articulated, and not subject to revision absent remarkable failures of proof.

One line of argument[2] pursued by Professor Bernstein follows from his observation that Daubert procedures are required to curtail litigation expert witness “adversarial bias.” Id. at 555. Bernstein traces adversarial bias to three sources:

(1) conscious bias;

(2) unconscious bias; and

(3) selection bias.

Id. Conscious bias stems from deliberate attempts by “hired guns” to deliver opinions that satisfy the lawyers who retained them. The problem of conscious bias is presented by “hired guns” who will adapt their opinions to the needs of the attorney who hires them. Unconscious biases are the more subtle, but no less potent determinants of expert witness behavior, which are created by financial dependence upon, and allegiance to, the witness’s paymaster. Selection bias results from lawyers’ ability to choose expert witnesses to support their claims, regardless whether those witnesses’ opinions are representative of the scientific community. Id.

Professor Bernstein’s taxonomy of bias is important, but incomplete. First, the biases he identifies operate fulsomely in regulatory settings. Although direct financial remuneration is usually not a significant motivation for a scientist to testify before an agency, or to submit a whitepaper, professional advancement and cause advocacy are often powerful incentives at work. These incentives for self-styled public interest zealots may well create more powerful distortions of scientific judgment than any monetary factors in private litigation settings. As for selection bias, lawyers are ethically responsible for screening their expert witnesses, and there can be little doubt that once expert witnesses are disclosed, their opinions will align with their sponsoring parties’ interests. This systematic bias, however, does not necessarily mean that both side’s expert witnesses will necessarily be unrepresentative or unscientific. In the silicone gel breast implant litigation (MDL 926), Judge Pointer, the presiding judge, insisted that both sides’ witnesses were “too extreme,” and he was stunned when his court-appointed expert witnesses filed reports that vindicated the defendants’ expert witnesses’ positions[3]. The defendants had selected expert witnesses who analyzed the data on sound scientific principles; the plaintiffs had selected expert witnesses who overreached in their interpretation of the evidence. Furthermore, many scientific disputes, which find their way into the courtroom, will not have the public profile of silicone gel breast implants, and for which there may be no body of scientific community opinion from which lawyers could select “outliers,” even if they wished to do so.

Professor Bernstein’s offered taxonomy of bias is incomplete because it does not include the most important biases that jurors (and many judges) struggle to evaluate:

random errors;

systematic biases;

confounding; and

cognitive biases.

These errors and biases, along with their consequential fallacies of reasoning, apply with equal force to agency and litigation science. Bernstein does point out, however, an important institutional difference between jury or judge trials and agency review and decisions based upon scientific evidence: agencies often have extensive in-house expertise. Although agency expertise may sometimes be blinded by its policy agenda, agency procedures usually afford the public and the scientific community to understand what the agency decided, and why, and to respond critically when necessary. In the case of the Food and Drug Administration, agency decisions, whether pro- or contra-industry positions are dissected and critiqued by the scientific and statistical community with great care and relish. Nothing of the same sort is possible in response to a jury verdict.

Professor Bernstein is not a science nihilist, and he would not have reviewing courts give a pass to whatever nonsense federal agencies espouse. He calls for enforcement of available statutory requirements that agency action be based upon the “best available science,” and for requiring agencies to explicitly separate and state their policy and scientific judgments. Bernstein also urges greater use of agency peer review, such as occasionally seen from the Institute of Medicine (soon to be the National Academy of Medicine), and the use of Daubert-like criteria for testimony at agency hearings. Bernstein at 554.

Proponents of regulatory Daubert should take Professor Bernstein’s essay to heart, with a daily dose of atorvastatin. Importing Rule 702 into agency proceedings may well undermine the rule’s import in litigation, civil and criminal, while achieving little in the regulatory arena. Consider the pending OSHA rulemaking for lowering the permissible exposure limit (PEL) of crystalline silica in the workplace. OSHA, and along with some public health organizations, has tried to justify this rulemaking on the basis of many overwrought claims of the hazards of crystalline silica exposure at current levels. Clearly, there are some workers who continue to work in unacceptably hazardous conditions, but the harms sustained by these workers can be tied to violations of the current PEL; they are hardly an argument for lowering that current PEL. Contrary to the OSHA’s parade of horribles, silicosis mortality in the United States has steadily declined over the last several decades. The following chart draws upon NIOSH and other federal governmental data:

 

Silicosis Deaths by Year

 

Silicosis deaths, crude and age-adjusted death rates, for U.S. residents age 15 and over, 1968–2007

from Susan E. Dudley & Andrew P. Morriss, “Will the Occupational Safety and Health Administration’s Proposed Standards for Occupational Exposure to Respirable Crystalline Silica Reduce Workplace Risk?” 35 Risk Analysis (2015), in press, doi: 10.1111/risa.12341 (NIOSH reference number: 2012F03–01, based upon multiple cause-of-death data from National Center for Health Statistics, National Vital Statistics System, with population estimates from U.S. Census Bureau).

The decline in silicosis mortality is all the more remarkable because it occurred in the presence of stimulated reporting from silicosis litigation, and misclassification of coal workers’ pneumoconiosis in coal-mining states.

The decline in silicosis mortality may be helpfully compared with the steady rise in mortality from accidental falls among men and women 65 years old, or older:

CDC MMWR Death Rates from Unintentional Falls 2015

Yahtyng Sheu, Li-Hui Chen, and Holly Hedegaard, “QuickStats: Death Rates* from Unintentional Falls† Among Adults Aged ≥ 65 Years, by Sex — United States, 2000–2013,” 64 CDC MMWR 450 (May 1, 2015). Over the observation period, these death rates roughly doubled in both men and women.

Is there a problem with OSHA rulemaking? Of course. The agency has gone off on a regulatory frolic and detour trying to justify an onerous new PEL, without any commitment to enforcing its current silica PEL. OSHA has invoked the prospect of medical risks, many of which are unproven, speculative, and remote, such as lung cancer, autoimmune disease, and kidney disease. The agency, however, is awash with PhDs, and I fear that Professor Bernstein is correct that the distortions of the science are not likely to be corrected by applying Rule 702 to agency factfinding. Courts, faced with the complex prediction models, with disputed medical claims made by agency and industry scientists, will do what they usually do, shrug and defer. And the blow back of the “judicially approved” agency science in litigation contexts will be a cure worse than the disease. At bottom, the agency twisting of science is driven by policy goals and considerations, which require public debate and scrutiny, sound executive judgment, with careful legislative oversight and guidance.

[1] Even under the very low evidentiary and procedural hurdles, federal agencies still manage to outrun their headlights on occasion. See, e.g., Industrial Union Department v. American Petroleum Institute, 448 U.S. 607 (1980) (The Benzene Case); Gulf South Insulation v. U.S. Consumer Product Safety Comm’n, 701 F.2d 1137 (5th Cir. 1983); Corrosion Proof Fittings v. EPA, 947 F2d 1201 (5th Cir 1991).

[2] See also David E. Bernstein, “The Misbegotten Judicial Resistance to the Daubert Revolution,” 89 Notre Dame L. Rev. 27, 31 (2013); David E. Bernstein, “Expert Witnesses, Adversarial Bias, and the (Partial) Failure of the Daubert Revolution,” 93 Iowa L. Rev. 451, 456–57 (2008).

[3] Judge Pointer was less than enthusiastic about performing any gatekeeping role. Unlike most of today’s MDL judges, he was content to allow trial judges in the transferor districts to decide Rule 702 and other pre-trial issues. See Note, “District Judge Takes Issue With Circuit Courts’ Application of Gatekeeping Role” 3 Federal Discovery News (Aug. 1997) (noting that Chief Judge Pointer had criticized appellate courts for requiring district judges to serve as gatekeepers of expert witness testimony).

Posted in Rule 702, Scientific Evidence, Silica | Comments Off on Professor Bernstein’s Critique of Regulatory Daubert

Science as Adversarial Process versus Group Think

May 7th, 2015

Climate scientists, at least those scientists who believe that climate change is both real and an existential threat to human civilization, have invoked their consensus as an evidentiary ground for political action. These same scientists have also used their claim of a consensus to shame opposing points of view (climate change skeptics) as coming from “climate change deniers.”

Consensus, or “general acceptance” as it is sometimes cast in legal discussions, is rarely more than nose counting. At best, consensus is a proxy for data quality and inferential validity. At worst, consensus is a manifestation of group think and herd mentality. Debates about climate change, as well as most scientific issues, would progress more dependably if there were more data, and less harrumphing about consensus.

Olah’s Nobel Speech

One Nobel laureate, Professor George Olah, explicitly rejected the kumbaya view of science and its misplaced emphasis on consensus and collaboration. In accepting his Nobel Prize in Chemistry, Olah emphasized the value of adversarial challenges in refining and establishing scientific discovery:

“Intensive, critical studies of a controversial topic always help to eliminate the possibility of any errors. One of my favorite quotation is that by George von Bekessy (Nobel Prize in Medicine, 1961).

‘[One] way of dealing with errors is to have friends who are willing to spend the time necessary to carry out a critical examination of the experimental design beforehand and the results after the experiments have been completed. An even better way is to have an enemy. An enemy is willing to devote a vast amount of time and brain power to ferreting out errors both large and small, and this without any compensation. The trouble is that really capable enemies are scarce; most of them are only ordinary. Another trouble with enemies is that they sometimes develop into friends and lose a good deal of their zeal. It was in this way the writer lost his three best enemies. Everyone, not just scientists, needs a few good enemies!’”

George A. Olah, “My Search for Carbocations and Their Role in Chemistry,” Nobel Lecture (Dec. 8, 1994), quoting George von Békésy, Experiments in Hearing 8 (N.Y. 1960); see also McMillan v. Togus Reg’l Office, Dep’t of Veterans Affairs, 294 F. Supp. 2d 305, 317 (E.D.N.Y. 2003) (“As in political controversy, ‘science is, above all, an adversary process.’”) (internal citation omitted).

Carl Sagan expressed similar views about the importance of skepticism in science :

“At the heart of science is an essential balance between two seemingly contradictory attitudes — an openness to new ideas, no matter how bizarre or counterintuitive they may be, and the most ruthless skeptical scrutiny of all ideas, old and new. This is how deep truths are winnowed from deep nonsense.”

Carl Sagan, The Demon-Haunted World: Science as a Candle in the Dark (1995); See also Cary Coglianese, “The Limits of Consensus,” 41 Environment 28 (April 1999).

Michael Crichton, no fan of Sagan, agreed at least on the principle:

“I want to . . . talk about this notion of consensus, and the rise of what has been called consensus science. I regard consensus science as an extremely pernicious development that ought to be stopped cold in its tracks. Historically, the claim of consensus has been the first refuge of scoundrels; it is a way to avoid debate by claiming that the matter is already settled. Whenever you hear the consensus of scientists agrees on something or other, reach for your wallet, because you’re being had.

Let’s be clear: the work of science has nothing whatever to do with consensus. Consensus is the business of politics. Science, on the contrary, requires only one investigator who happens to be right, which means that he or she has results that are verifiable by reference to the real world. In science consensus is irrelevant. What is relevant is [sic] reproducible results. The greatest scientists in history are great precisely because they broke with the consensus.  There is no such thing as consensus science. If it’s consensus, it isn’t science. If it’s science, it isn’t consensus. Period.34

Michael Crichton, “Lecture at California Institute of Technology: Aliens Cause Global Warming” (Jan. 17, 2003) (describing many examples of how “consensus” science historically has frustrated scientific progress).

Crystalline Silica, Carcinogenesis, and Faux Consensus

Clearly, there are times when consensus in science works against knowledge and data-driven inferences. Consider the saga of crystalline silica and lung cancer. Suggestions that silica causes lung cancer date back to the 1930s, but the suggestions were dispelled by data. The available data were evaluated by the likes of Wilhelm Heuper[1], Cuyler Hammond[2] (Selikoff’s go-to-epidemiologist), Gerrit Schepers[3], and Hans Weill[4]. Even etiologic fabulists, such as Kaye Kilburn, disclaimed any connection between silica or silicosis and lung cancer[5]. As recently as 1988, august international committees, writing for the National Institute of Occupational Safety and Health, acknowledged the evidentiary insufficiency of any claim that silica caused lung cancer[6].

IARC (1987)

So what happened to the “consensus”? A group of activist scientists, who disagreed with the consensus, sought to establish their own, new consensus. Working through the International Agency for Research on Cancer (IARC), these scientists were able to inject themselves into the IARC working group process, and gradually raise the IARC ranking of crystalline silica. In 1987, the advocate scientists were able to move the IARC to adopt a “limited evidence” classification for crystalline silica.

The term “limited evidence” is defined incoherently by the IARC as evidence that provides for a “credible” causal explanation, even though chance, bias, and confounding have not been adequately excluded. Despite the incoherent definition that giveth and taketh away, the 1987 IARC reclassification[7] into Group 2A had regulatory consequences that saw silica classified as a “regulatory carcinogen,” or a substance that was “reasonably anticipated to be a carcinogen.”

The advocates’ prophecy was self-fulfilling. In 1996, another working group of the IARC met in Lyon, France, to deliberate on the classification of crystalline silica. The 1996 working group agreed, by a close vote, to reclassify crystalline silica as a “known human carcinogen,” or a Group 1 carcinogen. The decision was accepted and reported officially in volume 68 of the IARC monographs, in 1997.

According to participants, the debate was intense and the vote close. Here is the description from one of the combatants;

“When the IARC Working Group met in Lyon in October 1996 to assess the carcinogenicity of crystalline silica, a seemingly interminable debate ensued, only curtailed by a reminder from the Secretariat that the IARC was concerned with the identification of carcinogenic hazards and not the evaluation of risks. The important distinction between the potential to cause disease in certain circumstances, and in what circumstances, is not always appreciated.

*   *   *   *   *

Even so, the debate in Lyon continued for some time, finally ending in a narrow vote, reflecting the majority view of the experts present at that particular time.”

See Corbett McDonald, “Silica and Lung Cancer: Hazard or Risk,” 44 Ann. Occup. Hyg. 1, 1 (2000); see also Corbett McDonald & Nicola Cherry, “Crystalline Silica and Lung Cancer: The Problem of Conflicting Evidence,” 8 Indoor Built Env’t 8 (1999).

Although the IARC reclassification hardly put the silica lung cancer debate to rest, it did push the regulatory agencies to walk in lockstep with the IARC and declare crystalline silica to be a “known human carcinogen.” More important, it gave regulators and scientists an excuse to avoid the hard business of evaluating complicated data, and of thinking for themselves.

Post IARC

From a sociology of science perspective, the aftermath of the 1997 IARC monograph is a fascinating natural experiment to view the creation of a sudden, thinly supported, new orthodoxy. To be sure, there were scientists who looked carefully at the IARC’s stated bases and found them inadequate, inconsistent, and incoherent[8]. One well-regarded pulmonary text in particular gives the IARC and regulatory agencies little deference:

“Silica-induced lung cancer

A series of studies suggesting that there might be a link between silica inhalation and lung cancer was reviewed by the International Agency for Research on Cancer in 1987, leading to the conclusion that the evidence for carcinogenicity of crystalline silica in experimental animals was sufficient, while in humans it was limited.112 Subsequent epidemiological publications were reviewed in 1996, when it was concluded that the epidemiological evidence linking exposure to silica to the risk of lung cancer had become somewhat stronger.113 but that in the absence of lung fibrosis remained scanty.113 The pathological evidence in humans is also weak in that premalignant changes around silicotic nodules are seldom evident.114 Nevertheless, on this rather insubstantial evidence, lung cancer in the presence of silicosis (but not coal or mixed-dust pneumoconiosis) has been accepted as a pre­scribed industrial disease in the UK since 1992.115 Some subsequent studies have provided support for this decision.116 In contrast to the sparse data on classic silicosis, the evidence linking carcinoma of the lung to the rare diffuse pattern of fibrosis attributed to silica and mixed dusts is much stronger and appears incontrovertible.33,92”

Bryan Corrin[9] & Andrew Nicholson, Pathology of the Lungs (3d ed. 2011).

=======================================================

Cognitive biases cause some people to see a glass half full, while others see it half empty. Add a “scientific consensus” to the mix, and many people will see a glass filled 5% as 95% full.

Consider a paper by Centers for Disease Control and NIOSH authors on silica exposure and morality from various diseases. Geoffrey M. Calvert, Faye L. Rice, James M. Boiano, J. W. Sheehy, and Wayne T. Sanderson, “Occupational silica exposure and risk of various diseases: an analysis using death certificates from 27 states of the United States,” 60 Occup. Envt’l Med. 122 (2003). The paper was nominated for the Charles Shepard Award for Best Scientific Publication by a CDC employee, and was published in the British Medical Journal’s publication on occupational medicine. The study analyzed death certificate data from the U.S. National Occupational Mortality Surveillance (NOMS) system, which is based upon the collaboration of NIOSH, the National Center for Health Statistics, the National Cancer Institute, and some state health departments. Id. at 122.

From about 4.8 million death certificates included in their analysis, the authors found a statistically decreased mortality odds ratio (MOR) for lung cancer among those who had silicosis (MOR = 0.70, 95% C.I., 0.55 to 0.89). Of course, with silicosis on the death certificates along with lung cancer, the investigators could be reasonably certain about silica exposure. Given the group-think in occupational medicine about silica and lung cancer, the authors struggled to explain away their finding:

“Although many studies observed that silicotics have an increased risk for lung cancer, a few studies, including ours, found evidence suggesting the lack of such an association. Although this lack of consistency across studies may be related to differences in study design, it suggests that silicosis is not necessary for an increased risk of lung cancer among silica exposed workers.”

Well this statement is at best disingenuous. The authors did not merely find a lack of an association; they found a statistically significance inverse or “negative” association between silicosis and lung cancer. So it is not the case that silicosis is not necessary for an increased risk; silicosis is antithetical to an increased risk.

Looking at only death certificate information, without any data on known or suspected confounders (“diet, hobbies, tobacco use, alcohol use, or medication,” id. at 126, or comorbid diseases or pulmonary impairment, or other occupational or environmental exposures), the authors inferred low, medium, high, and “super high” silica exposure from job categories. Comparing the ever-exposed categories with low exposure yielded absolutely no association between exposure and lung cancer, and subgroup analyses (without any correction for multiple comparisons) found little association, although two subgroups were nominally statistically significantly increased, and one was nominally statistically significantly decreased, at very small deviations from expected:

Lung Cancer Mortality Odds Ratios (p-value for trend < 0.001)

ever vs. low/no exposure:                0.99 (0.98 to 1.00)

medium vs. low/no exposure:         0.88 (0.87 to 0.90)

high vs. low/no exposure:                 1.13 (1.11 to 1.15)

super high vs. low/no exposure:      1.13 (1.06 to 1.21)

Id. at Table 4, and 124.

On this weak evidentiary display, the authors declare that their “study corroborates the association between crystalline silica exposure and silicosis, lung cancer.” Id. at 123. In their conclusions, they elaborate:

“Our findings support an association between high level crystalline silica exposure and lung cancer. The statistically significant MORs for high and super high exposures compared with low/no exposure (MORs = 1.13) are consistent with the relative risk of 1.3 reported in a meta-analysis of 16 cohort and case-control studies of lung cancer in crystalline silica exposed workers without silicosis”

Id. at 126. Actually not; Calvert’s reported MORs exclude an OR of 1.3.

The Calvert study thus is a stunning example of authors, prominent in the field of public health, looking at largely exculpatory data and declaring that they have confirmed an important finding of silica carcinogenesis. And to think that United States taxpayers paid for this paper, and that the authors almost received an honorific award for this thing!

[1] Wilhelm Hueper, “Environmental Lung Cancer,” 20 Industrial Medicine & Surgery 49, 55-56 (1951) (“However, the great majority of investigators have come to the conclusion that there does not exist any causal relation between silicosis and pulmonary or laryngeal malignancy”).

[2] Cuyler Hammond & W. Machle, “Environmental and Occupational Factors in the Development of Lung Cancer,” Ch. 3, pp. 41, 50, in E. Mayer & H. Maier, Pulmonary Carcinoma: Pathogenesis, Diagnosis, and Treatment (N.Y. 1956) (“Studies by Vorwald (41) and others agree in the conclusion that pneumoconiosis in general, and silicosis in particular, do not involve any predisposition of lung cancer.”).

[3] Gerrit Schepers, “Occupational Chest Diseases,” Chap. 33, in A. Fleming, et al., eds., Modern Occupational Medicine at 455 (Philadelphia 2d ed. 1960) (“Lung cancer, of course, occurs in silicotics and is on the increase. Thus far, however, statistical studies have failed to reveal a relatively enhanced incidence of pulmonary neoplasia in silicotic subjects.”).

[4] Ziskind, Jones, and Weill, “State of the Art: Silicosis” 113 Am. Rev. Respir. Dis. 643, 653 (1976) (“There is no indication that silicosis is associated with increased risk for the development of cancer of the respiratory or other systems.”); Weill, Jones, and Parkes, “Silicosis and Related Diseases, Chap. 12, in Occupational Lung Disorders (3d ed. 1994) (“It may be reasonably concluded that the evidence to date that occupational exposure to silica results in excess lung cancer risk is not yet persuasive.”).

[5] Kaye Kilburn, Ruth Lilis, Edwin Holstein, “Silicosis,” in Maxcy-Rosenau, Public Health and Preventive Medicine, 11th ed., at 606 (N.Y. 1980) (“Lung cancer is apparently not a complication of silicosis”).

[6] NIOSH Silicosis and Silicate Disease Committee, “Diseases Associated With Exposure to Silica and Nonfibrous Silicate Minerals,” 112 Arch. Path. & Lab. Med. 673, 711b, ¶ 2 (1988) (“The epidemiological evidence at present is insufficient to permit conclusions regarding the role of silica in the pathogenesis of bronchogenic carcinoma.”)

[7] 42 IARC Monographs on the Evaluation of the Carcinogenic Risk of Chemicals to Humans at 22, 111, § 4.4 (1987).

[8] See, e.g., Patrick A. Hessel, John F. Gamble, J. Bernard L. Gee, Graham Gibbs, Francis H. Y. Green, W. Keith C. Morgan, and Brooke T. Mossman, “Silica, Silicosis, and Lung Cancer: A Response to A Recent Working Group,” 42 J. Occup. Envt’l Med. 704, 718 (2000) (“The data demonstrate a lack of association between lung cancer and exposure to crystalline silica in human studies. Furthermore, silica is not directly genotoxic and has been to be a pulmonary carcinogen in only one animal species, the rat, which seems to be an inappropriate carcinogenesis in humans.”)

[9] Professor of Thoracic Pathology, National Heart and Lung Institute, Imperial College School of Medicine; Honorary Consultant Pathologist, Brompton Hospital, London, UK.

Posted in Causation, Cognitive Biases, Scientific Evidence | Comments Off on Science as Adversarial Process versus Group Think

ALI Reporters Are Snookered by Racette Fallacy

April 27th, 2015

In the Reference Manual on Scientific Evidence, the authors of the epidemiology chapter advance instances of acceleration of onset of disease as an example of a situation in which reliance upon doubling of risk will not provide a reliable probability of causation calculation[1]. In a previous post, I suggested that the authors’ assertion may be unfounded. See “Reference Manual on Scientific Evidence on Relative Risk Greater Than Two For Specific Causation Inference” (April 25, 2014). Several epidemiologic methods would permit the calculation of relative risk within specific time windows from first exposure.

The American Law Institute (ALI) Reporters, for the Restatement of Torts, make similar claims.[2] First, the Reporters, citing the Manual’s second edition, repeat the Manual’s claim that:

 “Epidemiologists, however, do not seek to understand causation at the individual level and do not use incidence rates in group to studies to determine the cause of an individual’s disease.”

American Law Institute, Restatement (Third) of Torts: Liability for Physical and Emotional Harm § 28(a) cmt. c(4) & rptrs. notes (2010) [Comment c(4)]. In making this claim, the Reporters ignore an extensive body of epidemiologic studies on genetic associations and on biomarkers, which do address causation implicitly or explicitly, on an individual level.

The Reporters also repeat the Manual’s doubtful claim that acceleration of onset of disease prevents an assessment of attributable risk, although they acknowledge that an average earlier age of onset would form the basis of damages calculations rather than calculations for damages for an injury that would not have occurred but for the tortious exposure. Comment c(4). The Reporters go a step further than the Manual, however, and provide an example of the acceleration-of-onset studies that they have in mind:

“For studies whose results suggest acceleration, see Brad A. Racette, Welding-Related Parkinsonism: Clinical Features, Treatments, and Pathophysiology,” 56 Neurology 8, 12 (2001) (stating that authors “believe that welding acts as an accelerant to cause [Parkinson’s Disease]… .”

The citation to Racette’s 2001 paper[3] is curious, interesting, disturbing, and perhaps revealing. In this 2001 paper, Racette misrepresented the type of study he claimed to have done, and the inferences he drew from his case series are invalid. Any one experienced in the field of epidemiology would have dismissed this study, its conclusions, and its suggested relation between welding and parkinsonism.

Dr. Brad A. Racette teaches and practices neurology at Washington University in St. Louis, across the river from a hotbed of mass tort litigation, Madison County, Illinois. In the 1990s, Racette received referrals from plaintiffs’ attorneys to evaluate their clients in litigation over exposure to welding fumes. Plaintiffs were claiming that their occupational exposures caused them to develop manganism, a distinctive parkinsonism that differs from Parkinson’s disease [PD], but has signs and symptoms that might be confused with PD by unsophisticated physicians unfamiliar with both manganism and PD.

After the publication of his 2001 paper, Racette became the darling of felon Dicky Scruggs and other plaintiffs’ lawyers. The litigation industrialists invited Racette and his team down to Alabama and Mississippi, to conduct screenings of welding tradesmen, recruited by Scruggs and his team, for potential lawsuits for PD and parkinsonism. The result was a paper that helped Scruggs propel a litigation assault against the welding industry.[4]

Racette’s 2001 paper was accompanied by a press release, as have many of his papers, in which he was quoted as stating that “[m]anganism is a very different disease” from PD. Gila Reckess, “Welding, Parkinson’s link suspected” (Feb. 9, 2001)[5].

Racette’s 2001 paper provoked a strongly worded letter that called Racette and his colleagues out for misrepresenting the nature of their work:

“The authors describe their work as a case–control study. Racette et al. ascertained welders with parkinsonism and compared their concurrent clinical features to those of subjects with PD. This is more consistent with a cross-sectional design, as the disease state and factors of interest were ascertained simultaneously. Cross-sectional studies are descriptive and therefore cannot be used to infer causation.”

*****

“The data reported by Racette et al. do not necessarily support any inference about welding as a risk factor in PD. A cohort study would be the best way to evaluate the role of welding in PD.”

Bernard Ravina, Andrew Siderowf, John Farrar, Howard Hurtig, “Welding-related parkinsonism: Clinical features, treatment, and pathophysiology,” 57 Neurology 936, 936 (2001).

As we will see, Dr. Ravina and his colleagues were charitable to suggest that the study was more compatible with a cross-sectional study. Racette had set out to determine “whether welding-related parkinsonism differs from idiopathic PD.” He claimed that he had “performed a case-control study,” with a case group of welders and two control groups. His inferences drawn from his “data” are, however, fallacious because he employed an invalid study design.

In reality, Racette’s paper was nothing more than a chart review, a case series of 15 “welders” in the context of a movement disorder clinic. After his clinical and radiographic evaluation, Racette found that these 15 cases were clinically indistinguishable from PD, and thus unlike manganism. Racette did not reveal whether any of these 15 welders had been referred by plaintiffs’ counsel; nor did he suggest that these welding tradesmen made up a disproportionate number of his patient base in St. Louis, Missouri.

Racette compared his selected 15 career welders with PD to his general movement disorders clinic patient population, for comparison. From the patient population, Racette deployed two “control” groups, one matched for age and sex with the 15 welders, and the other group not matched. The America Law Institute reporters are indeed correct that Racette suggested that the average age of onset for these 15 welders was lower than that for his non-welder patients, but their uncritical embrace overlooked the fact that Racette’s suggestion does not support his claimed inference that in welders, therefore, “welding exposure acts as an accelerant to cause PD.”

Racette’s claimed inference is remarkable because he did not perform an analytical epidemiologic study that was capable of generating causal inferences. His paper incongruously presents odds ratios, although the controls have PD, the disease of interest, which invalidates any analytical inference from his case series. Given the referral and selection biases inherent in tertiary-care specialty practices, this paper can provide no reliable inferences about associations or differences in ages of onset. Even within the confines of a case series misrepresented to be a case-control study, Racette acknowledged that “[s]ubsequent comparisons of the welders with age-matched controls showed no significant differences.”

NOT A CASE-CONTROL STUDY

That Racette wrongly identified his paper as a case-control study is beyond debate. How the journal Neurology accepted the paper for publication is a mystery. The acceptance of the inference by the ALI Reporter, lawyers and judges, is regrettable.

Structurally, Racette’s paper could never quality as a case-control study, or any other analytical epidemiologic study. Here is how a leading textbook on case-control studies defines a case-control study:

“In a case-control study, individuals with a particular condition or disease (the cases) are selected for comparison with a series of individuals in whom the condition or disease is absent (the controls).”

James J. Schlesselman, Case-control Studies. Design, Conduct, Analysis at 14 (N.Y. 1982)[6].

Every patient in Racette’s paper, welders and non-welders, have the outcome of interest, PD. There is no epidemiologic study design that corresponds to what Racette did, and there is no way to draw any useful inference from Racette’s comparisons. Racette’s paper violates the key principle for a proper case-control study; namely, all subjects must be selected independently of the study exposure that is under investigation. Schlesselman stressed that that identifying an eligible case or control must not depend upon that person’s exposure status for any factor under consideration. Id. Racette’s 2001 paper deliberately violated this basic principle.

Racette’s study design, with only cases with the outcome of interest appearing in the analysis, recklessly obscures the underlying association between the exposure (welding) and age in the population. We would, of course, expect self-identified welders to be younger than the average Parkinson’s disease patient because welding is physical work that requires good health. An equally fallacious study could be cobbled together to “show” that the age-of-onset of Parkinson’s disease for sitcom actors (such as Michael J. Fox) is lower than the age-of-onset of Parkinson’s disease for Popes (such as John Paul II). Sitcom actors are generally younger as a group than Popes. Comparing age of onset between disparate groups that have different age distributions generates a biased comparison and an erroneous inference.

The invalidity and fallaciousness of Racette’s approach to studying the age-of-onset issue of PD in welders, and his uncritical inferences, have been extensively commented upon in the general epidemiologic literature. For instance, in studies that compared the age at death for left-handed versus right-handed person, studies reported an observed nine-year earlier death for left handers, leading to (unfounded) speculation that earlier mortality resulted from birth and life stressors and accidents for left handers, living in a world designed to accommodate right-handed person[7]. The inference has been shown to be fallacious and the result of social pressure in the early twentieth century to push left handers to use their right hands, a prejudicial practice that abated over the decades of the last century. Left handers born later in the century were less likely to be “switched,” as opposed to those persons born earlier and now dying, who were less likely to be classified as left-handed, as a result of a birth-cohort effect[8]. When proper prospective cohort studies were conducted, valid data showed that left-handers and right-handers have equivalent mortality rates[9].

Epidemiologist Ken Rothman addressed the fallacy of Racette’s paper at some length in one of his books:

“Suppose we study two groups of people and look at the average age at death among those who die. In group A, the average age of death is 4 years; in group B, it is 28 years. Can we say that being a member of group A is riskier than being a member of group B? We cannot… . Suppose that group A comprises nursery school students and group B comprises military commandos. It would be no surprise that the average age at death of people who are currently military commandos is 28 years or that the average age of people who are currently nursery students is 4 years. …

In a study of factory workers, an investigator inferred that the factory work was dangerous because the average age of onset of a particular kind of cancer was lower in these workers than among the general population. But just as for the nursery school students and military commandos, if these workers were young, the cancers that occurred among them would have to be occurring in young people. Furthermore, the age of onset of a disease does not take into account what proportion of people get the disease.

These examples reflect the fallacy of comparing the average age at which death or disease strikes rather than comparing the risk of death between groups of the same age.”

Kenneth J. Rothman, “Introduction to Epidemiologic Thinking,” in Epidemiology: An Introduction at 5-6 (N.Y. 2002).

And here is how another author of Modern Epidemiology[10] addressed the Racette fallacy in a different context involving PD:

“Valid studies of age-at-onset require no underlying association between the risk factor and aging or birth cohort in the source population. They must also consider whether a sufficient induction time has passed for the risk factor to have an effect. When these criteria and others cannot be satisfied, age-specific or standardized risks or rates, or a population-based case-control design, must be used to study the association between the risk factor and outcome. These designs allow the investigator to disaggregate the relation between aging and the prevalence of the risk factor, using familiar methods to control confounding in the design or analysis. When prior knowledge strongly suggests that the prevalence of the risk factor changes with age in the source population, case-only studies may support a relation between the risk factor and age-at-onset, regardless of whether the inference is justified.”

Jemma B. Wilk & Timothy L. Lash, “Risk factor studies of age-at-onset in a sample ascertained for Parkinson disease affected sibling pairs: a cautionary tale,” 4 Emerging Themes in Epidemiology 1 (2007) (internal citations omitted) (emphasis added).

A properly designed epidemiologic study would have avoided Racette’s fallacy. A relevant cohort study would have enrolled welders in the study at the outset of their careers, and would have continued to follow them even if they changed occupations. A case-control study would have enrolled cases with PD and controls without PD (or more broadly, parkinsonism), with cases and controls selected independently of their exposure to welding fumes. Either method would have determined the rate of PD in both groups, absolutely or relatively. Racette’s paper, which completely lacked non-PD cases, could not have possibly accomplished his stated objectives, and it did not support his claims.

Racette’s questionable work provoked a mass tort litigation and ultimately federal Multi-District Litigation 1535.[11] Ultimately, analytical epidemiologic studies consistently showed no association between welding and PD. A meta-analysis published in 2012 ended the debate[12] as a practical matter, and MDL 1535 is no more. How strange that the ALI reporters chose the Racette work as an example of their claims about acceleration of onset!

[1] Michael D. Green, D. Michal Freedman, and Leon Gordis, “Reference Guide on Epidemiology,” in Federal Judicial Center, Reference Manual on Scientific Evidence 549, 614 (Wash., DC 3d ed. 2011).

[2] Michael D. Green was an ALI Reporter, and of course, an author of the chapter in the Reference Manual.

[3] Brad A. Racette, L. McGee-Minnich, S. M. Moerlein, J. W. Mink, T. O. Videen, and Joel S. Perlmutter, “Welding-related parkinsonism: clinical features, treatment, and pathophysiology,” 56 Neurology 8 (2001).

[4] See Brad A. Racette, S.D. Tabbal, D. Jennings, L. Good, Joel S. Perlmutter, and Brad Evanoff, “Prevalence of parkinsonism and relationship to exposure in a large sample of Alabama welders,” 64 Neurology 230 (2005); Brad A. Racette, et al., “A rapid method for mass screening for parkinsonism,” 27 Neurotoxicology 357 (2006) (duplicate publication of the earlier, 2005, paper).

[5] Previously available at <http://record.wustl.edu/archive/2001/02-09-01/articles/welding.html>, last visited on June 27, 2005.

[6] See also Brian MacMahon & Dimitrios Trichopoulos, Epidemiology. Principles and Methods at 229 (2ed 1996) (“A case-control study is an inquiry in which groups of individuals are selected based on whether they do (the cases) or do not (the controls) have the disease of which the etiology is to be studied.”); Jennifer L. Kelsey, W.D. Thompson, A.S. Evans, Methods in Observational Epidemiology at 148 (N.Y. 1986) (“In a case-control study, persons with a given disease (the cases) and persons without the disease (the controls) are selected … .”).

[7] See, e.g., Diane F. Halpern & Stanley Coren, “Do right-handers live longer?” 333 Nature 213 (1988); Diane F. Halpern & Stanley Coren, “Handedness and life span,” 324 New Engl. J. Med. 998 (1991).

[8] Kenneth J. Rothman, “Left-handedness and life expectancy,” 325 New Engl. J. Med. 1041 (1991) (pointing out that by comparing age of onset method, nursery education would be found more dangerous than paratrooper training, given that the age at death of pres-schoolers wo died would be much lower than that of paratroopers who died); see also Martin Bland & Doug Altman, “Do the left-handed die young?” Significance 166 (Dec. 2005).

[9] See Philip A. Wolf, Ralph B. D’Agostino, Janet L. Cobb, “Left-handedness and life expectancy,” 325 New Engl. J. Med. 1042 (1991); Marcel E. Salive, Jack M. Guralnik & Robert J. Glynn, “Left-handedness and mortality,” 83 Am. J. Public Health 265 (1993); Olga Basso, Jørn Olsen, Niels Holm, Axel Skytthe, James W. Vaupel, and Kaare Christensen, “Handedness and mortality: A follow-up study of Danish twins born between 1900 and 1910,” 11 Epidemiology 576 (2000). See also Martin Wolkewitz, Arthur Allignol, Martin Schumacher, and Jan Beyersmann, “Two Pitfalls in Survival Analyses of Time-Dependent Exposure: A Case Study in a Cohort of Oscar Nominees,” 64 Am. Statistician 205 (2010); Michael F. Picco, Steven Goodman, James Reed, and Theodore M. Bayless, “Methodologic pitfalls in the determination of genetic anticipation: the case of Crohn’s disease,” 134 Ann. Intern. Med. 1124 (2001).

[10] Kenneth J. Rothman, Sander Greenland, Timothy L. Lash, eds., Modern Epidemiology (3d ed. 2008).

[11] Dicky Scruggs served on the Plaintiffs’ Steering Committee until his conviction on criminal charges.

[12] James Mortimer, Amy Borenstein, and Lorene Nelson, “Associations of welding and manganese exposure with Parkinson disease: Review and meta-analysis,” 79 Neurology 1174 (2012).

Posted in Causation, Reference Manual on Scientific Evidence, Rule 702, Rule 703, Scientific Evidence | Comments Off on ALI Reporters Are Snookered by Racette Fallacy

Cherry Picking; Systematic Reviews; Weight of the Evidence

April 5th, 2015

In a paper prepared for one of Professor Margaret Berger’s symposia on law and science, Lisa Bero, a professor of clinical pharmacy in the University of California San Francisco’s School of Pharmacy identified a major source of error in published reviews of putative health effects:

“The biased citation of studies in a review can be a major source of error in the results of the review. Authors of reviews can influence their conclusions by citing only studies that support their preconceived, desired outcome.”

Lisa Bero, “Evaluating Systematic Reviews and Meta-Analyses,” 14 J. L. & Policy 569, 576 (2006). Biased citation, consideration, and reliance are major sources of methodological error in courtroom proceedings as well. Sometimes astute judges recognize and bar expert witnesses who would pass off their opinions, as well considered, when they are propped up only by biased citation. Unfortunately, courts have been inconsistent, sometimes rewarding cherry picking of studies by admitting biased opinions[1], sometimes unhorsing the would-be expert witnesses by excluding their opinions[2].

Given that cherry picking or “biased citation” is recognized in the professional community as a rather serious methodological sin, judges may be astonished to learn that both phrases, “cherry picking” and “biased citation” do not appear in the third edition of the Reference Manual on Scientific Evidence. Of course, the Manual could have dealt with the underlying issue of biased citation by affirmatively promoting the procedure of systematic reviews, but here again, the Manual falls short. There is no discussion of systematic review in the chapters on toxicology[3], epidemiology[4], or statistics[5]. Only the chapter on clinical medicine discusses the systematic review, briefly[6]. The absence of support for the procedures of systematic reviews, combined with the occasional cheerleading for “weight of the evidence,” in which expert witnesses subjectively include and weight studies to reach pre-ordained opinions, tends to undermines the reliability of the latest edition of the Manual[7].

[1] Spray-Rite Serv. Corp. v. Monsanto Co., 684 F.2d 1226, 1242 (7th Cir. 1982) (failure to consider factors identified by opposing side’s expert did not make testimony inadmissible).

[2] In re Zoloft, 26 F. Supp. 3d 449 (E.D. Pa. 2014) (excluding perinatal epidemiologist, Anick Bérard, for biased cherry picking of data points); In re Accutane, No. 271(MCL), 2015 WL 753674, 2015 BL 59277 (N.J.Super. Law Div. Atlantic Cty. Feb. 20, 2015) (excluding opinions Drs. Arthur Kornbluth and David Madigan because of their authors’ unjustified dismissal of studies that contradicted or undermined their opinions); In re Bextra & Celebrex Mktg. Sales Practices & Prods. Liab. Litig., 524 F.Supp. 2d 1166, 1175–76, 1179 (N.D.Cal.2007) (holding that expert witnesses may not ‘‘cherry-pick[ ]’’ observational studies to support a conclusion that is contradicted by randomized controlled trials, meta-analyses of such trials, and meta-analyses of observational studies; excluding expert witness who ‘‘ignores the vast majority of the evidence in favor of the few studies that support her conclusion’’); Grant v. Pharmative, LLC, 452 F. Supp. 2d 903, 908 (D. Neb. 2006) (excluding expert witness opinion testimony that plaintiff’s use of black cohash caused her autoimmune hepatitis) (“Dr. Corbett’s failure to adequately address the body of contrary epidemiological evidence weighs heavily against admission of his testimony.”); Downs v. Perstorp Components, Inc., 126 F. Supp. 2d 1090,1124-29 (E.D. Tenn. 1999) (expert’s opinion raised seven “red flags” indicating that his testimony was litigation biased), aff’d, 2002 U.S. App. Lexis 382 (6th Cir. Jan. 4, 2002).

[3] Bernard D. Goldstein & Mary Sue Henifin, “Reference Guide on Toxicology,” in Reference Manual on Scientific Evidence 633 (3d ed. 2011).

[4] Michael D. Green, D. Michal Freedman, and Leon Gordis, “Reference Guide on Epidemiology,” in Reference Manual on Scientific Evidence 549 (3d ed. 2011).

[5] David H. Kaye & David A. Freedman, “Reference Guide on Statistics,” in Reference Manual on Scientific Evidence 209 (3d ed. 2011).

[6] John B. Wong, Lawrence O. Gostin, and Oscar A. Cabrera, “Reference Guide on Medical Testimony,” in Federal Judicial Center and National Research Council, Reference Manual on Scientific Evidence 687 (3d ed. 2011).

[7] See Margaret A. Berger, “The Admissibility of Expert Testimony,” in Reference Manual on Scientific Evidence 11, 20 & n.51 (3d ed. 2011) (posthumously citing Milward v. Acuity Specialty Products Group, Inc., 639 F.3d 11, 26 (1st Cir. 2011), with approval, for reversing exclusion of expert witnesses who advanced “weight of the evidence” opinions).

Posted in Expert Witnesses, Reference Manual on Scientific Evidence, Scientific Evidence | Comments Off on Cherry Picking; Systematic Reviews; Weight of the Evidence

Johnson of Accutane – Keeping the Gate in the Garden State

March 28th, 2015

Flag of Aquitaine     Nelson Johnson is the author of Boardwalk Empire: The Birth, High Times, and Corruption of Atlantic City (2010), a rattling good yarn, which formed the basis for a thinly fictionalized story of Atlantic City under the control of mob boss (and Republican politician) Enoch “Nucky” Johnson. HBO transformed Johnson’s book into a multi-season series, with Steve Buscemi playing Nucky Johnson (Thompson in the series). Robert Strauss, “Judge Nelson Johnson: Atlantic City’s Godfather — A Q&A with Judge Nelson Johnson,” New Jersey Monthly (Aug. 16, 2010).

Nelson Johnson is also known as the Honorable Nelson Johnson, a trial court judge in Atlantic County, New Jersey, where he inherited some of the mass tort docket of Judge Carol Higbee. Judge Higbee has since ascended to the Appellate Division of the New Jersey Superior Court. One of the litigations Judge Johnson presides over is the mosh pit of isotretinoin (Accutane) cases, involving claims that the acne medication causes irritable bowel syndrome (IBS) and Crohn’s disease (CD). Judge Johnson is not only an accomplished writer of historical fiction, but he is also an astute evaluator of the facts and data, and the accompanying lawyers’ rhetoric, thrown about in pharmaceutical products liability litigation.

Perhaps more than his predecessor ever displayed, Judge Johnson recently demonstrated his aptitude for facts and data in serving as a gatekeeper of scientific evidence, as required by the New Jersey Supreme Court, in Kemp v. The State of New Jersey, 174 NJ 412 (2002). Faced with a complex evidentiary display on the validity and reliability of the scientific evidence, Judge Johnson entertained extensive briefings, testimony, and oral argument. When the dust settled, the court ruled that the proffered testimony of Dr, Arthur Kornbluth and Dr. David Madigan did not meet the liberal New Jersey test for admissibility. In re Accutane, No. 271(MCL), 2015 WL 753674, 2015 BL 59277 (N.J.Super. Law Div. Atlantic Cty. Feb. 20, 2015). And in settling the dust, Judge Johnson dispatched several bogus and misleading “lines of evidence,” which have become standard ploys to clog New Jersey and other courthouses.

Case Reports

As so often is the case when there is no serious scientific evidence of harm in pharmaceutical cases, plaintiffs in the Accutane litigation relied heavily upon case and adverse event reports. Id. at *11. Judge Johnson was duly unimpressed, and noted that:

“[u]nsystematic clinical observations or case reports and adverse event reports are at the bottom of the evidence hierarchy.”

Id. at *16.

Bootstrapped, Manufactured Evidence

With respect to case reports that are submitted to the FDA’s Adverse Event Reporting System (FAERS), Judge Johnson acknowledged the “serious limitations” of the hearsay anecdotes that make up such reports. Despite the value of AERs in generating signals for future investigation, Judge Johnson, citing FDA’s own description of the reporting system, concluded that the system’s anecdotal data are “not evidentiary in a court of law.” Id. at 14 (quoting FDA’s description of FAERS).

Judge Johnson took notice of another fact; namely, the industry litigation creates evidence that it then uses to claim causal connections in the courtroom. Plaintiffs’ lawyers in pharmaceutical cases routinely file Medwatch adverse event reports, which thus inflate the “signal,” they claim supports the signal of harm from medication use. This evidentiary bootstrapping machine was hard at work in the isotretinoin litigation. See Derrick J. Stobaugh, Parakkal Deepak, and Eli D. Ehrenpreis, “Alleged Isotretinoin-Associated Inflammatory Bowel Disease: Disproportionate reporting by attorneys to the Food and Drug Administration Adverse Event Reporting System,” 69 J. Am. Acad. Dermatol. 398 (2013) (“Attorney-initiated reports inflate the pharmacovigilance signal of isotretinoin-associated IBD in the FAERS.”). Judge Johnson gave a wry hat tip to plaintiffs’ counsel’s industry, by acknowledging that the litigation industry itself had inflated this signal-generating process:

“The legal profession is a bulwark of our society, yet the courts should never underestimate the resourcefulness of some attorneys.”

In re Accutane, 2015 WL 753674, at *15.

Bias and Confounding

The epidemiologic studies referenced by the parties had identified a fairly wide range of “risk factors” for irritable bowel syndrome, including many prevalent factors in Westernized countries such as prior appendectomy, breast-feeding as an infant, stress, Vitamin D deficiency, tobacco or alcohol use, refined sugars, dietary animal fat, fast food. In re Accutane, 2015 WL 753674, at *9. The court also noted that there were four medications known to be risk factors for IBD: aspirin, nonsteroidal anti-inflammatory medications (NSAIDs), oral contraceptives, and antibiotics.

In reviewing the plaintiffs’ expert witnesses’ methodology, Judge Johnson found that they had been inordinately, and inappropriately selective in the studies chosen for reliance. The challenged witnesses had discounted and discarded most of the available studies in favor of two studies that were small, biased, and not population based. Indeed, one of the studies evidenced substantial selection bias by using referrals to obtain study participants, a process deprecated by the trial court as “cherry picking the subjects.” Id. at *18. “The scientific literature does not support reliance upon such insignificant studies to arrive at conclusions.” Id.

Animal Studies

Both sides in the isotretinoin cases seemed to concede the relative unimportance of animal studies. The trial court discussed the limitations on animal studies, especially the absence of a compelling animal model of human irritable bowel syndrome. Id. at *18.

Cherry Picking and Other Crafty Stratagems

With respect to the complete scientific evidentiary display, plaintiffs asserted that their expert witnesses had considered everything, but then failed to account for most of the evidence. Judge Johnson found this approach deceptive and further evidence of a cherry-picking, pathological methodology:

‘‘Finally, coursing through Plaintiffs’ presentation is a refrain that is a ruse. Repeatedly, counsel for the Plaintiffs and their witnesses spoke of ‛lines of evidence”, emphasizing that their experts examined ‛the same lines of evidence’ as did the experts for the Defense. Counsels’ sophistry is belied by the fact that the examination of the ‘lines of evidence’ by Plaintiffs’ experts was highly selective, looking no further than they wanted to—cherry picking the evidence—in order to find support for their conclusion-driven testimony in support of a hypothesis made of disparate pieces, all at the bottom of the medical evidence hierarchy.’’

Id. at *21.

New Jersey Rule of Evidence 703

The New Jersey rules of evidence, like the Federal Rules, imposes a reasonableness limit on what sorts of otherwise inadmissible evidence an expert witness may rely upon. See “RULE OF EVIDENCE 703 — Problem Child of Article VII” (Sept. 9, 2011). Although Judge Johnson did not invoke Rule 703 specifically, he was clearly troubled by plaintiffs’ expert witnesses’ reliance upon an unadjusted odds ratio from an abstract, which did not address substantial confounding from a known causal risk factor – antibiotics use. Judge Johnson concluded that the reliance upon the higher, unadjusted risk figure, contrary to the authors’ own methods and conclusions, and without a cogent explanation for so doing was “pure advocacy” on the part of the witnesses. In re Accutane, 2015 WL 753674, at *17; see also id. at *5 (citing Landrigan v. Celotex Corp., 127 N.J. 404, 417 (1992), for the proposition that “when an expert relies on such data as epidemiological studies, the trial court should review the studies, as well as other information proffered by the parties, to determine if they are of a kind on which such experts ordinarily rely.”).

Discordance Between Courtroom and Professional Opinions

One of plaintiffs’ expert witnesses, Dr. Arthur Kornbluth actually had studied putative association between isotretinoin and CD before he became intensively involved in litigation as an expert witness. In re Accutane, 2015 WL 753674, at *7. Having an expert witness who is a real world expert can be a plus, but not when that expert witness maintains a double standard for assessing causal connections. Back in 2009, Kornbluth published an article, “Ulcerative Colitis Practice Guidelines in Adults” in The American Journal of Gastroenterology. Id. at *10. This positive achievement became a large demerit when cross-examination at the Kemp hearing revealed that Kornbluth had considered but rejected the urgings of a colleague, Dr. David Sachar, to comment on isotretinoin as a cause of irritable bowel syndrome. In front of Judge Johnson, Dr. Kornbluth felt no such scruples. Id. at *11. Dr. Kornbluth’s stature in the field of gastroenterology, along with his silence on the issue in his own field, created a striking contrast with his stridency about causation in the courtroom. The contrast raised the trial court’s level of scrutiny and skepticism about his causal opinions in the New Jersey litigation. Id. (citing and quoting Soldo v. Sandoz Pharms. Corp, 244 F. Supp. 2d 434, 528 (W.D. Pa. 2003) (“Expert opinions generated as the result of litigation have less credibility than opinions generated as the result of academic research or other forms of ‘pure’ research.”) (“The expert’s motivation for his/her study and research is important. … We may not ignore the fact that a scientist’s normal work place is the lab or field, not the courtroom or the lawyer’s office.”).

Meta-Analysis

Meta-analysis has become an important facet of pharmaceutical and other products liability litigation[1]. Fortunately for Judge Johnson, he had before him an extremely capable expert witness, Dr. Stephen Goodman, to explain meta-analysis generally, and two meta-analyses performed on isotretinoin and irritable bowel outcomes. In re Accutane, 2015 WL 753674, at *8. Dr. Goodman explained that:

“the strength of the meta-analysis is that no one feature, no one study, is determinant. You don’t throw out evidence except when you absolutely have to.”

Id. Dr. Goodman further explained that plaintiffs’ expert witnesses’ failure to perform a meta-analysis was telling meta-analysis “can get us closer to the truth.” Id.

Some Nitpicking

Specific Causation

After such a commanding judicial performance by Judge Johnson, nitpicking on specific causation might strike some as ungrateful. For some reason, however, Judge Johnson cited several cases on the appropriateness of expert witnesses’ reliance upon epidemiologic studies for assessing specific causation or for causal apportionment between two or more causes. In re Accutane, 2015 WL 753674, at *5 (citing Landrigan v. Celotex Corp., 127 N.J. 404 (1992), Caterinicchio v. Pittsburgh Corning, 127 N.J. 428 (1992), and Dafler v. Raymark Inc., 259 N.J. Super. 17, 36 (App. Div. 1992), aff’d. o.b. 132 N.J. 96 (1993)). Fair enough, but specific causation was not at issue in the Accutane Kemp hearing, and the Landrigan and Caterinicchio cases are irrelevant to general causation.

In both Landrigan and Caterincchio, the defendants moved for directed verdicts by arguing that, assuming arguendo that asbestos causes colon cancer, the plaintiffs’ expert witnesses had not presented a sufficient opinion to support that Landrigan’s and Caterinnichio’s colon cancers were caused by asbestos. See “Landrigan v. The Celotex Corporation, Revisited” (June 4, 2013). General causation was thus never at issue, and the holdings never addressed the admissibility of the expert witnesses’ causation opinions. Only sufficiency of the opinions that equated increased risks, less than 2.0, to specific causation was at issue in the directed verdicts, and the appeals taken from the judgments entered on those verdicts.

Judge Johnson, in discussing previous case law suggests that the New Jersey Supreme Court reversed and remanded the Landrigan case for trial, holding that “epidemiologists could help juries determine causation in toxic tort cases and rejected the proposition that epidemiological studies must show a relative risk factor of 2.0 before gaining acceptance by a court.” In re Accutane, 2015 WL 753674, at *5, citing Landrigan, 127 N.J. at 419. A close and fair reading of Landrigan, however, shows that it was about a directed verdict, 127 N.J. at 412, and not a challenge to the use of epidemiologic studies generally, or to their use to show general causation.

Necessity of Precise Biological Mechanism

In the Accutane hearings, the plaintiffs’ counsel and their expert witnesses failed to provide a precise biological mechanism of the cause of IBD. Judge Johnson implied that any study that asserted that Accutane caused IBD ‘‘would, of necessity, require an explication of a precise biological mechanism of the cause of IBD and no one has yet to venture more than alternate and speculative hypotheses on that question.’’ In re Accutane, 2015 WL 753674, at *8. Conclusions of causality, however, do not always come accompanied by understood biological mechanisms, and Judge Johnson demonstrated that the methods and evidence relied upon by plaintiffs’ expert witnesses could not, in any event, allow them to draw causal conclusions.

Interpreting Results Contrary to Publication Authors’ Interpretations

There is good authority, no less than the United States Supreme Court in Joiner, that there is something suspect in expert witnesses’ interpreting a published study’s results in contrary to the authors’ publication. Judge Johnson found that the plaintiffs’ expert witnesses in the Accutane litigation had inferred that two studies showed increased risk when the authors of those studies had concluded that their studies did not appear to show an increased risk. Id. at *17. There will be times, however, when a published study may have incorrectly interpreted its own data, when “real” expert witnesses can, and should, interpret the data appropriately. Accutane was not such a case. In In re Accutane, Judge Johnson carefully documented and explained how the plaintiffs’ expert witnesses’ supposed reinterpretation was little more than attempted obfuscation. His Honor concluded that the witnesses’ distortion of, and ‘‘reliance upon these two studies is fatal and reveals the lengths to which legal counsel and their experts are willing to contort the facts and torture the logic associated with Plaintiffs’ hypothesis.’’ Id. at *18.

[1] “The Treatment of Meta-Analysis in the Third Edition of the Reference Manual on Scientific Evidence” (Nov. 14, 2011) (The Reference Manual fails to come to grips with the prevalence and importance of meta-analysis in litigation, and fails to provide meaningful guidance to trial judges).

Posted in Causation, Expert Witnesses, Rule 702, Rule 703, Scientific Evidence | Comments Off on Johnson of Accutane – Keeping the Gate in the Garden State

The Mythology of Linear No-Threshold Cancer Causation

March 13th, 2015

“For the great enemy of the truth is very often not the lie—deliberate, contrived, and dishonest—but the myth—persistent, persuasive, and unrealistic. Too often we hold fast to the clichés of our forebears. We subject all facts to a prefabricated set of interpretations. We enjoy the comfort of opinion without the discomfort of thought.”

John F. Kennedy, Yale University Commencement (June 11, 1962)

         *        *        *        *        *        *        *        *        *

The linear no-threshold model for risk assessment has its origins in a dubious attempt of scientists playing at policy making[1]. The model has survived as a political strategy to inject the precautionary principle into regulatory decision making, but it has turned into a malignant myth in litigation over low-dose exposures to putative carcinogens. Ignorance or uncertainty about low-dose exposures is turned into an affirmative opinion that the low-dose exposures are actually causative. Call it contrived, or dishonest, or call it a myth, the LNT model is an intellectual cliché.

The LNT cliché pervades American media as well as courtrooms. Earlier this week, the New York Times provided a lovely example of the myth taking center stage, without explanation or justification. Lumber Liquidators is under regulatory and litigation attack for having sold Chinese laminate wood flooring made with formaldehyde-containing materials. According to a “60 Minutes” investigation, the flooring off-gases formaldehyde at concentrations in excess of regulatory permissible levels. See Aaron M. Kessler & Rachel Abrams, “Homeowners Try to Assess Risks From Chemical in Floors,” New York Times (Mar. 10, 2015).

The Times reporters, in discussing whether a risk exists to people who live in houses and apartments with the Lumber Liquidators flooring sought out and quoted the opinion of Marilyn Howarth:

“Any exposure to a carcinogen can increase your risk of cancer,” said Marilyn Howarth, a toxicologist at the University of Pennsylvania’s Perelman School of Medicine.

Id. Dr. Howarth, however, is not a toxicologist; she is an occupational and environmental physician, and serves as the Director of Occupational and Environmental Consultation Services at the Hospital of the University of Pennsylvania. She is also an adjunct associate professor of emergency medicine, and the Director, of the Community Outreach and Engagement Core, Center of Excellence in Environmental Toxicology, at the University of Pennsylvania Perelman School of Medicine. Without detracting from Dr. Howarth’s fine credentials, the New York Times reporters might have noticed that Dr. Howarth’s publications are primarily on latex allergies, and not on the issue of the effect of low-dose exposure to carcinogens.

The point is not to diminish Dr. Howarth’s accomplishments, but to criticize the Times reporters for seeking out an opinion of a physician whose expertise is not well matched to the question they raise about risks, and then to publish that opinion even though it is demonstrably wrong. Clearly, there are some carcinogens, and perhaps all, that do not increase risk at “any exposure.” Consider ethanol, which is known to cause cancer of the larynx, liver, female breast, and perhaps other organs[2]. Despite known causation, no one would assert that “any exposure” to alcohol-containing food and drink increases the risk of these cancers. And the same could be said for most, if not all, carcinogens. The human body has defense mechanisms to carcinogens, including DNA repair mechanisms and programmed cell suicide, which work to prevent carcinogenesis from low-dose exposures.

The no threshold hypothesis is really at best an hypothesis, with affirmative evidence showing that the hypothesis should be rejected for some cancers[3]. The factual status of LNT is a myth; it is an opinion, and a poorly supported opinion at that.

         *        *        *        *        *        *        *        *        *

“There are, in fact, two things: science and opinion. The former brings knowledge, the latter ignorance.”

Hippocrates of Cos

[1] See Edward J. Calabrese, “Cancer risk assessment foundation unraveling: New historical evidence reveals that the US National Academy of Sciences (US NAS), Biological Effects of Atomic Radiation (BEAR) Committee Genetics Panel falsified the research record to promote acceptance of the LNT,” 89 Arch. Toxicol. 649 (2015); Edward J. Calabrese & Michael K. O’Connor, “Estimating Risk of Low Radiation Doses – A Critical Review of the BEIR VII Report and its Use of the Linear No-Threshold (LNT) Hypothesis,” 182 Radiation Research 463 (2014); Edward J. Calabrese, “Origin of the linearity no threshold (LNT) dose–response concept,” 87 Arch. Toxicol. 1621 (2013); Edward J. Calabrese, “The road to linearity at low doses became the basis for carcinogen risk assessment,” 83 Arch. Toxicol. 203 (2009).

[2] See, e.g., IARC Monographs on the Evaluation of Carcinogenic Risks to Humans – Alcohol Consumption and Ethyl Carbamate; volume 96 (2010).

[3] See, e.g., Jerry M. Cuttler, “Commentary on Fukushima and Beneficial Effects of Low Radiation,” 11 Dose-Response 432 (2013); Jerry M. Cuttler, “Remedy for Radiation Fear – Discard the Politicized Science,” 12 Dose Response 170 (2014).

Posted in Causation, Risk and Risk Factor, Scientific Evidence | Comments Off on The Mythology of Linear No-Threshold Cancer Causation

« Older Entries
Newer Entries »