TORTINI

For your delectation and delight, desultory dicta on the law of delicts.

Litigation Science – In re Zambelli-Weiner

April 8th, 2019

Back in 2001, in the aftermath of the silicone gel breast implant litigation, I participated in a Federal Judicial Center (FJC) television production of “Science in the Courtroom, program 6” (2001). Program six was a round-table discussion among the directors (past, present, and future) of the FJC, all of whom were sitting federal judges, with two lawyers in private practice, Elizabeth Cabraser and me.1 One of the more exasperating moments in our conversation came when Ms. Cabraser, who represented plaintiffs in the silicone litigation, complained that Daubert was unfair because corporate defendants were able to order up supporting scientific studies, whereas poor plaintiffs counsel did not have the means to gin up studies that confirmed what they knew to be true.2 Refraining from talking over her required all the self-restraint I could muster, but I did eventually respond by denying her glib generalization and offering the silicone litigation as one in which plaintiffs, plaintiffs’ counsel, and plaintiffs’ support groups were all involved in funding and directing some of the sketchiest studies, most of which managed to find homes in so-called peer-reviewed journals of some sort, even if not the best.

The litigation connections of the plaintiff-sponsored studies in the silicone litigation were not apparent on the face of the published articles. The partisan funding and provenance of the studies were mostly undisclosed and required persistent discovery and subpoenas. Cabraser’s propaganda reinforced the recognition of what so-called mass tort litigation had taught me about all scientific studies: “trust but verify.” Verification is especially important for studies that are sponsored by litigation-industry actors who have no reputation at stake in the world of healthcare.

Verification is not a straightforward task, however. Peer-review publication usually provides some basic information about “methods and materials,” but rarely if ever do published articles provide sufficient data and detail about methodology to replicate the reported analysis. In legal proceedings, verification of studies conducted and relied upon by testifying expert witnesses is facilitated by the rules of expert witness discovery. In federal court, expert witnesses must specify all opinions and all bases for their opinions. When such witnesses rely upon their own studies, and thus have had privileged access to the complete data and all analyses, courts have generally permitted full inquiry into the underlying materials of relied-upon studies. On the other, when the author of a relied-upon study is a “stranger to the litigation,” neither a party nor a retained expert witness, courts have permitted generally more limited discovery of the study’s full data set and analyses. Regardless of the author’s status, the question remains how litigants are to challenge an adversary’s expert witness’s trusted reliance upon a study, which cannot be “verified.”

Most lawyers would prefer, of course, to call an expert witness who has actually conducted studies pertinent to the issues in the case. The price, however, of allowing the other side to discover the underlying data and materials of the author expert witness’s studies may be too high. The relied-upon studies may well end up discredited, as well as the professional reputation of the expert witness. The litigation industry has adapted to these rules of discovery by avoiding, in most instances, calling testifying expert witnesses who have published studies that might be vulnerable.3

One work-around to the discovery rules lies in the use of “consulting, non-testifying expert witnesses.” The law permits the use of such expert witnesses to some extent to facilitate candid consultations with expert witnesses, usually without concerns that communications will be shared with the adversary party and witnesses. The hope is that such candid communications will permit realistic assessment of partisan positions, as well as allowing scientists and scholars to participate in an advisory capacity without the burden of depositions, formal report writing, and appearances at judicial hearings and trials. The confidentiality of consulting expert witnesses is open to abuse by counsel who would engage the consultants to conduct and publish studies, which can then be relied upon by the testifying expert witnesses. The upshot is that legal counsel can manipulate the published literature in a favorable way, without having to disclose their financial sponsorship or influence of the published studies used by their testifying expert witnesses.

This game of hiding study data and sponsorship through the litigation industry’s use of confidential consulting expert witnesses pervades so-called mass tort litigation, which provides ample financial incentives for study sponsorship and control. Defendants will almost always be unable to play the game, without detection. A simple interrogatory or other discovery request about funding of studies will reveal the attempt to pass off a party-sponsored study as having been conducted by disinterested scientists. Furthermore, most scientists will feel obligated to reveal corporate funding as a potential conflict of interest, in their submission of manuscripts for publication.

Revealing litigation-industry (plaintiffs’) funding of studies is more complicated. First, the funding may be through one firm, which is not the legal counsel in the case for which discovery is being conducted. In such instances, the plaintiff’s lawyers can truthfully declare that they lack personal knowledge of any financial support for studies relied upon by their testifying expert witnesses. Second, the plaintiffs’ lawyer firm is not a party is not itself subject to discovery. Even if the plaintiffs’ lawyers funded a study, they can claim, with plausible deniability, that they funded the study in connection with another client’s case, not the client who is plaintiff in the case in which discovery is sought. Third, the plaintiffs’ firm may take the position, however dubious it might be, that the funding of the relied-upon study was simply a confidential consultation with the authors of that study, and not subject to discovery.

The now pending litigation against ondansetron (Zofran) provides the most recent example of the dubious use of consulting expert witnesses to hide party sponsorship of an epidemiologic study. The plaintiffs, who are claiming that Zofran causes birth defects in this multi-district litigation assigned to Judge F. Dennis Saylor, have designated Dr. Carol Luik as their sole testifying expert witness on epidemiology. Dr. Luik, in turn, has relied substantially upon a study conducted by Dr. April Zambelli-Weiner.4

According to motion papers filed by defendants,5 the plaintiffs’ counsel initially claimed that they had no knowledge of any financial support or conflicts for Dr Zambelli-Weiner. The conflict-of-interest disclosure in Zambelli-Weiner’s paper was, to say the least, suspicious:

The authors declare that there was no outside involvement in study design; in the collection, analysis and interpretation of data; in the writing of the manuscript; and in the decision to submit the manuscript for publication.”

As an organization TTi reports receiving funds from plaintiff law firms involved in ondansetron litigation and a manufacturer of ondansetron.”

According to its website, TTi

is an economically disadvantaged woman-owned small business headquartered in Westminster, Maryland. We are focused on the development, evaluation, and implementation of technologies and solutions that advance the transformation of data into actionable knowledge. TTi serves a diverse clientele, including all stakeholders in the health space (governments, payors, providers, pharmaceutical and device companies, and foundations) who have a vested interest in advancing research to improve patient outcomes, population health, and access to care while reducing costs and eliminating health disparities.”

According to defendants’ briefing, and contrary to plaintiffs’ initial claims and Zambelli-Weiner’s anemic conflicts disclosure, plaintiffs’ counsel eventually admitted that “Plaintiffs’ Leadership Attorneys paid $210,000 as financial support relating to” Zambelli-Weiner’s epidemiologic study. The women at TTi are apparently less economically disadvantaged than advertised.

The Zofran defendants served subpoenas duces tecum and ad testificandum on two of the study authors, Drs. April Zambelli-Weiner and Russell Kirby. Curiously, the plaintiffs (who would seem to have no interest in defending the third-party subpoenas) sought a protective order by arguing that defendants were harassing “third-party scientists.” Their motion for protection conveniently and disingenuously omitted, that Zambelli-Weiner had been a paid consultant to the Zofran plaintiffs.

Judge Saylor refused to quash the subpoenas, and Zambelli-Weiner appeared herself, through counsel, to seek a protective order. Her supporting affidavit averred that she had not been retained as an expert witness, and that she had no documents “concerning any data analyses or results that were not reported in the [published study].” Zambelli-Weiner’s attempt to evade discovery was embarrassed by her having presented a “Zofran Litigation Update” with Plaintiffs’ counsel Robert Jenner and Elizabeth Graham at a national conference for plaintiffs’ attorneys. Judge Saylor was not persuaded, and the MDL court refused Dr. Zambelli-Weiner’s motion. The law and the public has a right to every man’s, and every woman’s, (even if economically disadvantaged) evidence.6

Tellingly, in the aftermath of the motions to quash, Zambelli-Weiner’s counsel, Scott Marder, abandoned his client by filing an emergency motion to withdraw, because “certain of the factual assertions in Dr. Zambelli-Weiner’s Motion for Protective Order and Affidavit were inaccurate.” Mr. Marder also honorably notified defense counsel that he could no longer represent that Zambelli-Weiner’s document production was complete.

Early this year, on January 29, 2019, Zambelli-Weiner submitted, through new counsel, a “Supplemental Affidavit,” wherein she admitted she had been a “consulting expert” witness for the law firm of Grant & Eisenhofer on the claimed teratogenicity of Zofran.7 Zambelli-Weiner also produced a few extensively redacted documents. On February 1, 2019, Zambelli-Weiner testified at deposition that the moneys she received from Grant & Eisenhofer were not to fund her Zofran study, but for other, “unrelated work.” Her testimony was at odds with the plaintiffs’ counsel’s confession that the $210,000 related to her Zofran study.

Zambelli-Weiner’s etiolated document production was confounded by the several hundred of pages of documents produced by fellow author, Dr. Russell Kirby. When confronted with documents from Kirby’s production, Zambelli-Weiner’s lawyer unilaterally suspended the deposition.

Deja Vu All Over Again

Federal courts have seen the Zambelli maneuver before. In litigation over claimed welding fume health effects, plaintiffs’ counsel Richard (Dickie) Scruggs and colleagues funded some neurological researchers to travel to Alabama and Mississippi to “screen” plaintiffs and potential plaintiffs in litigation for over claims of neurological injury and disease from welding fume exposure, with a novel videotaping methodology. The plaintiffs’ lawyers rounded up the research subjects (a.k.a. clients and potential clients), talked to them before the medical evaluations, and administered the study questionnaires. The study subjects were clearly aware of Mr. Scruggs’ “research” hypothesis, and had already promised him 40% of any recovery.8

After their sojourn, at Scruggs’ expense to Alabama and Mississippi, the researchers wrote up their results, with little or no detail of the circumstances of how they had acquired their research “participants,” or those participants’ motives to give accurate or inaccurate medical and employment history information.9

Defense counsel served subpoenas upon both Dr. Racette and his institution, Washington University St. Louis, for the study protocol, underlying data, data codes, and all statistical analyses. Racette and Washington University resisted sharing their data and materials with every page in the Directory of Non-Transparent Research. They claimed that the subpoenas sought production of testimony, information and documents in violation of:

(1) the Federal Regulations set forth in the Department of Health and Human Services Policy for Protection of Human Research Subjects,

(2) the Federal regulations set forth in the HIPPA Regulations,

(3) the physician/patient privilege,

(4) the research scholar’s privilege,

(5) the trade secret/confidential research privilege and

(6) the scope of discovery as codified by the Federal Rules of Civil Procedure and the Missouri Rules of Civil Procedure.”

After a long discovery fight, the MDL court largely enforced the subpoenas.10 The welding MDL court ordered Racette to produce

a ‘limited data set’ which links the specific categories requested by defendants: diagnosis, occupation, and age. This information may be produced as a ‘deidentified’ data set, such that the categories would be linked to each particular patient, without using any individual patient identifiers. This data set should: (1) allow matching of each study participant’s occupational status and age with his or her neurological condition, as diagnosed by the study’s researchers; and (2) to the greatest extent possible (except for necessary de-identification), show original coding and any code-keys.”

After the defense had the opportunity to obtain and analyze the underlying data in the Scruggs-Racette study, the welding plaintiffs retreated from their epidemiologic case. Various defense expert witnesses analyzed the underlying data produced by Racette, and prepared devastating rebuttal reports. These reports were served upon plaintiffs’ counsel, whose expert witnesses never attempted any response. Reliance upon Racette’s study was withdrawn or abandoned. After the underlying data were shared with the parties to MDL 1535, no scientist appeared to defend the results in the published papers.11 The Racette Alabama study faded into the background of the subsequent welding-fume cases and trials.

The motion battle in the welding MDL revealed interesting contradictions, similar to those seen in the Zambelli-Weiner affair. For example, Racette claimed he had no relationship whatsoever with plaintiffs’ counsel, other than showing up by happenstance in Alabama at places where Scruggs’ clients also just happened to show up. Racette claimed that the men and women he screened were his patients, but he had no license to practice in Alabama, where the screenings took place. Plaintiffs’ counsel disclaimed that Racette was a treating physician, which acknowledgment would have made the individual’s screening results discoverable in their individual cases. And more interestingly, plaintiffs’ counsel claimed that both Dr. Racette and Washington University were “non-testifying, consulting experts utilized to advise and assist Plaintiffs’ counsel with respect to evaluating and assessing each of their client’s potential lawsuit or claim (or not).”12

Over the last decade or so, best practices and codes of conduct for the relationship between pharmacoepidemiologists and study funders have been published.13 These standards apply with equal force to public agencies, private industry, and regulatory authories. Perhaps it is time for them to specify that the apply to the litigation industry as well.


1 See Smith v. Wyeth-Ayerst Labs. Co., 278 F. Supp. 2d 684, 710 & n. 56 (W.D.N.C. 2003).

2 Ironically, Ms. Cabraser has published her opinion that failure to disclose conflicts of interest and study funding should result in evidentiary exclusions, a view which would have simplified and greatly shortened the silicone gel breast implant litigation. See Elizabeth J. Cabraser, Fabrice Vincent & Alexandra Foote, “Ethics and Admissibility: Failure to Disclose Conflicts of Interest in and/or Funding of Scientific Studies and/or Data May Warrant Evidentiary Exclusions,” Mealey’s Emerging Drugs Reporter (Dec. 2002).

3 Litigation concerning Viagra is one notable example where plaintiffs’ counsel called an expert witness who was the author of the very study that supposedly supported their causal claim. It did not go well for the plaintiffs or the expert witness. See Lori B. Leskin & Bert L. Slonim, “A Primer on Challenging Peer-Reviewed Scientific Literature in Mass Tort and Product Liability Actions,” 25 Toxics L. Rptr. 651 (Jul. 1, 2010).

4 April Zambelli‐Weiner, Christina Via, Matt Yuen, Daniel Weiner, and Russell S. Kirby, “First Trimester Pregnancy Exposure to Ondansetron and Risk of Structural Birth Defects,” 83 Reproductive Toxicology 14 (2019).

5 Nate Raymond, “GSK accuses Zofran plaintiffs’ law firms of funding academic study,” Reuters (Mar. 5, 2019).

6 See Branzburg v. Hayes, 408 U.S. 665, 674 (1972).

7 Affidavit of April Zambelli-Weiner, dated January 9, 2019 (Doc. No. 1272).

8 The plaintiffs’ lawyers’ motive and opportunity to poison the study by coaching their “clients” was palpable. See David B. Resnik & David J. McCann, “Deception by Research Participants,” 373 New Engl. J. Med. 1192 (2015).

9 See Brad A. Racette, S.D. Tabbal, D. Jennings, L. Good, J.S. Perlmutter, and Brad Evanoff, “Prevalence of parkinsonism and relationship to exposure in a large sample of Alabama welders,” 64 Neurology 230 (2005); Brad A. Racette, et al., “A rapid method for mass screening for parkinsonism,” 27 Neurotoxicology 357 (2006) (a largely duplicative report of the Alabama welders study).

10 See, e.g., In re Welding Fume Prods. Liab. Litig., MDL 1535, 2005 WL 5417815 (N.D. Ohio Oct. 18, 2005) (upholding defendants’ subpoena for protocol, data, data codes, statistical analyses, and other things from Dr. Racette’s Alabama study on welding and parkinsonism).

11 Racette sought and obtained a protective order for the data produced, and thus I still cannot share the materials he provided asking that any reviewer sign the court-mandated protective order. Revealingly, Racette was concerned about who had seen his underlying data, and he obtained a requirement in the court’s non-disclosure affidavit that any one who reviews the underlying data will not sit on peer review of his publications or his grant applications. See Motion to Compel List of Defendants’ Reviewers of Data Produced by Brad A. Racette, M.D., and Washington University Pursuant to Protective Order, in In re Welding Fume Products Liab. Litig., MDL No. 1535, Case 1:03-cv-17000-KMO, Document 1642-1 (N.D. Ohio Feb. 14, 2006). Curiously, Racette never moved to compel a list of Plaintiffs’ Reviewers!

12 Plaintiffs’ Motion for Protective Order, Motion to Reconsider Order Requiring Disclovery from Dr. Racette, and Request for In Camera Inspection as to Any Responses or Information Provided by Dr. Racette, filed in Solis v. Lincoln Elec. Co., case No. 1:03-CV-17000, MDL 1535 (N.D. Ohio May 8, 2006).

13 See, e.g., Xavier Kurz, Susana Perez‐Gutthann, and the ENCePP Steering Group, “Strengthening standards, transparency, and collaboration to support medicine evaluation: Ten years of the European Network of Centres for Pharmacoepidemiology and Pharmacovigilance (ENCePP),” 27 Pharmacoepidem. & Drug Safety 245 (2018).

Two Stanford Researchers Are Anti-Semantic

August 4th, 2018

Two Stanford University communications researchers have shown that fraudulent publications and authors’ linguistic obfuscation are correlated, p < 0.05. David M. Markowitz & Jeffrey T. Hancock, “Linguistic Obfuscation in Fraudulent Science,” 35 J. Language & Social Psych. 435 (2016); Bjorn Carey, “Stanford researchers uncover patterns in how scientists lie about their data,Stanford Report (Nov. 16, 2015) [Carey, below]

Stanford Professor of Communication, Jeff Hancock, and graduate student David Markowitz observed that there are repeating patterns of expression in the language used by scientific fraudfeasors. They hypothesized that scientific fraudfeasors would signal their duplicity in their linguistic expressions as well. These authors created a linguistic obfuscation index based upon the prevalence of jargon, abstraction, positive emotion terms, and readability. They then compared the obfuscation index scores of 253 papers retracted for fraudulent data with 253 unretracted papers, and 63 papers retracted for reasons other than fraud. Not surprisingly, Hancock and Markowitz found differences, with fraudulent papers having higher obfuscation scores, and generally more jargon.

As Markowitz explained:

We believe the underlying idea behind obfuscation is to muddle the truth. *** Scientists faking data know that they are committing a misconduct and do not want to get caught. Therefore, one strategy to evade this may be to obscure parts of the paper. We suggest that language can be one of many variables to differentiate between fraudulent and genuine science.”

Carey. Professor Hancock acknowledged that there remained a high error rate in their obfuscation analysis, which needed to be lowered before automatic linguistic analyses could be useful for detecting fraud. Hancock also acknowledged that such the use of such a computerized linguistic tool might undermine the trust upon which science is based. Id.

Well detecting fraud might undermine trust, but look where trust has gotten us in science.

Trust but verify.

I cannot wait until I proffer the first expert witness rebuttal report in litigation, to show that my adversary’s expert witness has crossed the obfuscation line.

Infante-lizing the IARC

May 13th, 2018

Peter Infante, a frequently partisan, paid expert witness for the Lawsuit Industry, recently published a “commentary” in the red journal, the American Journal of Industrial Medicine, about the evils of scientists with economic interests commenting upon the cancer causation pronouncements of the International Agency for Research on Cancer (IARC). Peter F. Infante, Ronald Melnick, James Huff & Harri Vainio, “Commentary: IARC Monographs Program and public health under siege by corporate interests,” 61 Am. J. Indus. Med. 277 (2018). Infante’s rant goes beyond calling out scientists with economic interests on IARC working groups; Infante would silence all criticism of IARC pronouncements by anyone, including scientists, who has an economic interest in the outcome of a scientific debate. Infante points to manufacturing industry’s efforts to “discredit” recent IARC pronouncements on glyphosate and red meat, by which he means that there were scientists who had the temerity to question IARC’s processes and conclusions.

Apparently, Infante did not think his bias was showing or would be detected. He and his co-authors invoke militaristic metaphors to claim that the IARC’s monograph program, and indeed all of public health, is “under siege by corporate interests.” A farcical commentary at that, coming from such stalwarts of the Lawsuit Industry. Infante lists his contact information as “Peter F. Infante Consulting, LLC, Falls Church, Virginia,” and co-author Ronald Melnick can be found at “Ronald Melnick Consulting, LLC, North Logan, Utah.” A search on Peter Infante in Westlaw yields 141 hits, all on the plaintiffs’ side of health effects disputes; he is clearly no stranger to the world of litigation. Melnick is, to be sure, harder to find, but he does show up as a signatory on Raphael Metzger’s supposed amicus briefs, filed by Metzger’s litigation front organization, Council for Education and Research on Toxics.1

Of the commentary’s authors, only James Huff, of “James Huff Consulting, Durham, North Carolina,” disclosed a connection with litigation, as a consultant to plaintiffs on animal toxicology of glyphosate. Huff’s apparent transparency clouded up when it came to disclosing how much he has been compensated for his consulting activities for claimants in glyphosate litigation. In the very next breath, in unison, the authors announce unabashedly that “[a]ll authors report no conflicts of interest.” Infante at 280.

Of course, reporting “no conflicts of interest” does not mean that the authors have no conflicts of interest, financial, positional, and idealogical. Their statement simply means that they have not reported any conflicts, through inadvertence, willfulness, or blindness. The authors, and the journal, are obviously content to mislead their readership by not-so-clever dodges.

The clumsiness of the authors’ inability to appreciate their very own conflicts infects their substantive claims in this commentary. These “consultants” tell us solemnly that IARC “[m]eetings are openly transparent and members are vetted for conflicts of interest.” Infante at 277. Working group members, however, are vetted but only for manufacturing industry conflicts, not for litigation industry conflicts or for motivational conflicts, such as advancing their own research agendas. Not many scientists have a research agenda to show that chemicals do not cause cancer.

At the end of this charade, the journal provides additional disclosures [sic]. As for “Ethics and Approval Consent,” we are met with a bold “Not Applicable.” Indeed; ethics need not apply. Perhaps, the American Journal of Industrial Medicine is beyond good and evil. The journal’s “Editor of Record,” Steven B. Markowitz “declares that he has no conflict of interest in the review and publication decision regarding this article.” This is, of course, the same Markowitz who testifies frequently for the Lawsuit Industry, without typically disclosing this conflict on his own publications.

This commentary is yet another brushback pitch, which tries to chill manufacturing industry and scientists from criticizing the work of agencies, such as IARC, captured by lawsuit industry consultants. No one should be fooled other than Mother Jones.


1See, e.g., Ramos v. Brenntag Specialties, Inc., 372 P.3d 200 (Calif. 2016) (where plaintiff was represented by Metzger, and where CERT filed an amicus brief by the usual suspects, plaintiffs’ expert witnesses, including Melnick).

Statistical Deontology

March 2nd, 2018

In courtrooms across America, there has been a lot of buzzing and palavering about the American Statistical Association’s Statement on Statistical Significance Testing,1 but very little discussion of the Society’s Ethical Guidelines, which were updated and promulgated in the same year, 2016. Statisticians and statistics, like lawyers and the law, receive their fair share of calumny over their professional activities, but the statistician’s principal North American professional organization is trying to do something about members’ transgressions.

The American Statistical Society (ASA) has promulgated ethical guidelines for statisticians, as has the Royal Statistical Society,2 even if these organizations lack the means and procedures to enforce their codes. The ASA’s guidelines3 are rich with implications for statistical analyses put forward in all contexts, including in litigation and regulatory rule making. As such, the guidelines are well worth studying by lawyers.

The ASA Guidelines were prepared by the Committee on Professional Ethics, and approved by the ASA’s Board in April 2016. There are lots of “thou shall” and “thou shall nots,” but I will focus on the issues that are more likely to arise in litigation. What is remarkable about the Guidelines is that if followed, they probably are more likely to eliminate unsound statistical practices in the courtroom than the ASA State on P-values.

Defining Good Statistical Practice

Good statistical practice is fundamentally based on transparent assumptions, reproducible results, and valid interpretations.” Guidelines at 1. The Guidelines thus incorporate something akin to the Kumho Tire standard that an expert witness ‘‘employs in the courtroom the same level of intellectual rigor that characterizes the practice of an expert in the relevant field.’’ Kumho Tire Co. v. Carmichael, 526 U.S. 137, 152 (1999).

A statistician engaged in expert witness testimony should provide “only expert testimony, written work, and oral presentations that he/she would be willing to have peer reviewed.” Guidelines at 2. “The ethical statistician uses methodology and data that are relevant and appropriate, without favoritism or prejudice, and in a manner intended to produce valid, interpretable, and reproducible results.” Id. Similarly, the statistician, if ethical, will identify and mitigate biases, and use analyses “appropriate and valid for the specific question to be addressed, so that results extend beyond the sample to a population relevant to the objectives with minimal error under reasonable assumptions.” Id. If the Guidelines were followed, a lot of spurious analyses would drop off the litigation landscape, regardless whether they used p-values or confidence intervals, or a Bayesian approach.

Integrity of Data and Methods

The ASA’s Guidelines also have a good deal to say about data integrity and statistical methods. In particular, the Guidelines call for candor about limitations in the statistical methods or the integrity of the underlying data:

The ethical statistician is candid about any known or suspected limitations, defects, or biases in the data that may impact the integrity or reliability of the statistical analysis. Objective and valid interpretation of the results requires that the underlying analysis recognizes and acknowledges the degree of reliability and integrity of the data.”

Guidelines at 3.

The statistical analyst openly acknowledges the limits of statistical inference, the potential sources of error, as well as the statistical and substantive assumptions made in the execution and interpretation of any analysis,” including data editing and imputation. Id. The Guidelines urge analysts to address potential confounding not assessed by the study design. Id. at 3, 10. How often do we see these acknowledgments in litigation-driven analyses, or in peer-reviewed papers, for that matter?

Affirmative Actions Prescribed

In the aid of promoting data and methodological integrity, the Guidelines also urge analysts to share data when appropriate without revealing the identities of study participants. Statistical analysts should publicly correct any disseminated data and analyses in their own work, as well as working to “expose incompetent or corrupt statistical practice.” Of course, the Lawsuit Industry will call this ethical duty “attacking the messenger,” but maybe that’s a rhetorical strategy based upon an assessment of risks versus benefits to the Lawsuit Industry.

Multiplicity

The ASA Guidelines address the impropriety of substantive statistical errors, such as:

[r]unning multiple tests on the same data set at the same stage of an analysis increases the chance of obtaining at least one invalid result. Selecting the one “significant” result from a multiplicity of parallel tests poses a grave risk of an incorrect conclusion. Failure to disclose the full extent of tests and their results in such a case would be highly misleading.”

Guidelines at 9.

There are some Lawsuit Industrialists who have taken comfort in the pronouncements of Kenneth Rothman on corrections for multiple comparisons. Rothman’s views on multiple comparisons are, however, much broader and more nuanced than the Industry’s sound bites.4 Given that Rothman opposes anything like strict statistical significance testing, it follows that he is relatively unmoved for the need for adjustments to alpha or the coefficient of confidence. Rothman, however, has never deprecated the need to consider the multiplicity of testing, and the need for researchers to be forthright in disclosing the the scope of comparisons originally planned and actually done.


2 Royal Statistical Society – Code of Conduct (2014); Steven Piantadosi, Clinical Trials: A Methodologic Perspective 609 (2d ed. 2005).

3 Shelley Hurwitz & John S. Gardenier, “Ethical Guidelines for Statistical Practice: The First 60 Years and Beyond,” 66 Am. Statistician 99 (2012) (describing the history and evolution of the Guidelines).

4 Kenneth J. Rothman, “Six Persistent Research Misconceptions,” 29 J. Gen. Intern. Med. 1060, 1063 (2014).