TORTINI

For your delectation and delight, desultory dicta on the law of delicts.

Subgroups — Subpar Statistical Practice versus Fraud

July 24th, 2014

Several people have asked me why I do not enable comments on this blog.  Although some bloggers (e.g., Deborah Mayo’s Error Statistics site) have had great success in generating interesting and important discussions, I have seen too much spam on other websites, and I want to avoid having to police the untoward posts.  Still, I welcome comments and I try to respond to helpful criticism.  If and when I am wrong, I will gladly eat my words, which usually have been quite digestible.

Probably none of the posts here have generated more comments and criticisms than those written about the prosecution of Dr. Harkonen.  In general, critics have argued that defending Harkonen and his press release was tantamount to condoning bad statistical practice.  I have tried to show that Dr. Harkonen’s press release was much more revealing than it was portrayed in abbreviated accounts of his case, and the evidentiary support for his claim of efficacy in a subgroup was deeper and broader than acknowledged. The criticism and condemnation of Dr. Harkonen’s press release in the face of prevalent statistical practice, among leading journals and practitioners, is nothing short of hypocrisy and bad faith. If Dr. Harkonen deserves prison time for a press release, which promised a full analysis and discussion in upcoming conference calls and presentations at scientific meetings, then we can only imagine what criminal sanction awaits the scientists and journal editors who publish purportedly definitive accounts of clinical trials and epidemiologic studies, with subgroup analyses not prespecified and not labeled as post-hoc.

The prevalence of the practice does not transform Dr. Harkonen’s press release into “best practice,” but some allowance must be made for offering a causal opinion in the informal context of a press release rather than in a manuscript for submission to a journal.  And those critics, with prosecutorial temperaments, must recognize that, when the study was presented at conferences, and when manuscript was written up and submitted to the New England Journal of Medicine, the authors did reveal the ad hoc nature of the subgroup.

The Harkonen case will remain important for several reasons. There is an important distinction in the Harkonen case, ignored and violated by the government’s position, between opinion and fact.  If Harkonen is guilty of Wire Fraud, then so are virtually every cleric, minister, priest, rabbi, imam, mullah, and other religious person who makes supernatural claims and predictions.  Add in all politicians, homeopaths, vaccine deniers, and others who reject evidence for superstition, who are much more culpable than a scientist who accurately reports the actual data and p-value.

Then there is the disconnect between what expert witnesses are permitted to say and what resulted in Dr. Harkonen’s conviction. If any good could come from the government’s win, it would be the insistence upon “best practice” for gatekeeping of expert witness opinion testimony.

For better or worse, scientists often describe post-hoc subgroup findings as “demonstrated” effects. Although some scientists would disagree with this reporting, the practice is prevalent.  Some scientists would go further and contest the claim that pre-specified hypotheses are inherently more reliable than post-hoc hypotheses. See Timothy Lash & Jan Vandenbroucke, “Should Preregistration of Epidemiologic Study Protocols Become Compulsory?,” 23 Epidemiology 184 (2012).

One survey compared grant applications with later published papers and found that subgroup analyses were pre-specified in only a minority of cases; in a substantial majority (77%) of the subgroup analyses in the published papers, the analyses were not characterized as either pre-specified or post hoc. Chantal W. B. Boonacker, Arno W. Hoes, Karen van Liere-Visser, Anne G. M. Schilder, and Maroeska M. Rovers, “A Comparison of Subgroup Analyses in Grant

Applications and Publications,” 174 Am. J. Epidem. 291, 291 (2011).  Indeed, this survey’s comparison between grant applications and published papers revealed that most of the published subgroup analyses were post hoc, and that the authors of the published papers rarely reported justifications for their post-hoc subgroup. Id.

Again, for better or worse, the practice of presenting unplanned subgroup analyses, is common in the biomedical literature. Several years ago, the New England Journal of Medicine reported a survey of publication practice in its own pages, with findings similar to those of Boonacker and colleagues. Rui Wang, Stephen W. Lagakos, James H. Ware, David J. Hunter, and Jeffrey M. Drazen, “Statistics in Medicine — Reporting of Subgroup Analyses in Clinical Trials,” 357 New Eng. J. Med. 2189 (2007).  In general, Wang, et al.,  were unable to determine the total number of subgroup analyses performed; and in the majority (68%) of trials discussed, Wang could not determine whether the subgroup analyses were prespecified. Id. at 2912. Although Wang proposed guidelines for identifying subgroup analyses as prespecified or post-hoc, she emphasized that the proposals were not “rules” that could be rigidly prescribed. Id. at 2194.

The Wang study is hardly unique; the Journal of the American Medical Association reported a similar set of results. An-Wen Chan, Asbjørn Hrobjartsson, Mette T. Haahr, Peter C. Gøtzsche, and Douglas G. Altman, “Empirical Evidence for Selective Reporting of Outcomes in Randomized Trials Comparison of Protocols to Published Articles,” 291 J. Am. Med. Ass’n 2457 (2004).  Chan and colleagues set out to document and analyze “outcome reporting bias” in studies; that is, the extent to which publications fail to report accurately the pre-specified outcomes in published studies of randomized clinical trials.  The authors compared and analyzed protocols and published reports of randomized clinical trials conducted in Denmark in 1994 and 1995. Their findings document a large discrepancy between idealized notion of pre-specification of study design, outcomes, and analyses, and the actual practice revealed by later publication.

Chan identified 102 clinical trials, with 3,736 outcomes, and found that 50% of efficacy, and 65% of harm outcomes were incompletely reported. There was a statistically significant risk of statistically significant outcomes to be fully reported compared with statistically insignificant results. (pooled odds ratio for efficacy outcomes = 2.4; 95% confidence interval, 1.4 – 4.0, and pooled odds ratio for harm outcomes = 4.7; 95% confidence interval, 1.8 -12.0. Their comparison of protocols with later published articles revealed that a majority of trials (62%) had at least one primary outcome that was changed, omitted, or innovated in the published version. The authors concluded that published accounts of clinical trials were frequently incomplete, biased, and inconsistent with protocols.

This week, an international group of scientists published their analysis of agreement vel non between protocols and corresponding later publications of randomized clinical trials. Matthias Briel, DISCO study group, “Subgroup analyses in randomised controlled trials: cohort study on trial protocols and journal publications,” 349 Brit. Med. J. g4539 (Published 16 July 2014). Predictably, the authors found a good deal of sloppy practice, or worse.  Of the 515 journal articles identified, about half (246 or 47.8%) reported one or more subgroup analysis. Of the articles that reported subgroup analyses, 81 (32.9%) publications stated that the subgroup analyses were prespecified, but in 28 of these articles (34.6%), the corresponding protocols did not identify the subgroup analysis.

In 86 of the publications surveyed, the authors found that the articles claimed a subgroup “effect,” but only 36 of the corresponding protocols reported a planned subgroup analysis.  Briel and the DISCO study group concluded that protocols of randomized clinical trials insufficiently describe subgroup analyses. In over one-third of publications, the articles reported subgroup analyses not pre-specified in earlier protocols. The DISCO study group called for access to protocols and statistical analysis plans for all randomized clinical trials.

In view of these empirical data, the government’s claims against Dr. Harkonen stand out, at best, as vindictive, selective prosecution.

Careless Scholarship About Silica History

July 21st, 2014

David Egilman is the Editor in Chief of the International Journal of Occupational and Environmental Health (IJOEH). A YouTube “selfie” interview provides some insight into Dr. Egilman’s motivations and editorial agenda.  Previous posts have chronicled Egilman’s testimonial adventures because of his propensity to surface in litigations of interest. See, e.g., “David Egilman’s Methodology for Divining Causation” (Sept. 6, 2012); “Egilman Petitions the Supreme Court for Review of His Own Exclusion in Newkirk v. Conagra Foods” (Dec. 13, 2012).

Dr. Egilman has used his editorial role at the IJOEH to disseminate his litigation positions.  Several of articles are little more than his litigation reports, filed in various cases, ranging from occupational dust disease claims to pharmaceutical off-target effect claims. A recent issue of the IJOEH has yet another article of this ilk, which scatters invective across several litigations. David Egilman, Tess Bird[1], and Caroline Lee[2], “Dust diseases and the legacy of corporate manipulation of science and law, 20 Internat’l J. Occup. & Envt’l Health 115 (2014).

The article mostly concerns Egilman’s allegations that companies influenced the scientific, medical, and governmental understanding and perception of asbestos hazards.  I will defer to others to address his allegations with respect to asbestos. The article, however, in its Abstract, takes broader aim at other exposures, in particular, silica:

“Knowledge that asbestos and silica were hazardous to health became public several decades after the industry knew of the health concerns. This delay was largely influenced by the interests of Metropolitan Life Insurance Company (MetLife) and other asbestos mining and product manufacturing companies.”

Egilman at 115, Abstract.

In their Abstract, the authors further claim their purpose

“To understand the ongoing corporate influence on the science and politics of asbestos and silica exposure, including litigation defense strategies related to historical manipulation of science.”

Egilman at 115. I demur for the time being with respect to asbestos, but the authors’ claims about silica are never supported in their article. A brief review of two monographs by Frederick L. Hoffman should be sufficient to condemn the authors’ carelessness to the dustbin of occupational history. Frederick L. Hoffman, Mortality from Respiratory Diseases in the Dusty Trades; Dep’t of Labor, Bureau of Labor Statistics (1918); The Problem of Dust Phthisis in the Granite Stone Industry; Dep’t of Labor, Bureau of Labor Statistics (1922).  The bibliographies in both these monographs documents the widespread interest in, and awareness of, the occupational hazards of silica dusts, going back into the 19th century, among the media, the labor movement, and the non-industrial scientific community.

Not surprisingly, the authors’ conclusions are stated only in terms of asbestos hazards, knowledge, and company conduct:

“Conclusions: Asbestos product companies would like the public to believe that there was a legitimate debate surrounding the dangers of asbestos during the twentieth century, particularly regarding the link to cancer, which delayed adequate regulation. The asbestos–cancer link was not a legitimate contestation of science; rather the companies directly manipulated the scientific literature. There is evidence that industry manipulation of scientific literature remains a continuing problem today, resulting in inadequate regulation and compensation and perpetuating otherwise preventable worker and consumer injuries and deaths.”

The authors note that Rutherford Johnstone’s 1960 “seminal” textbook relied upon a study (Braun and Truan), which study Egilman attacks as corrupted by industry influence. Rutherford Johnstone & Seward E. Miller, Occupational diseases and industrial medicine 328 (Philadelphia 1960). According to the Egilman, Rutherford Johnstone was the official American Medical Asociation’s consultant for occupational disease questions, which explains why he was providing answers to questions submitted to the Journal of the American Medical Association, on silica and asbestos issues. The authors note that Johnstone, in 1961, asserted that there was no epidemiologic evidence that asbestos causes lung cancer among American workers, which view reflects Johnstone’s reliance upon the Braun-Truan study. The authors fail, however, to note that Johnstone also opined that

“There is no epidemiological evidence that silicosis, resulting from undue exposure to free silica produces cancer of the lung.”

Rutherford T. Johnstone, “Silicosis and Cancer,” 176 J. Am. Med. Ass’n 81, 81 (1961). Neither the authors nor anyone else has ever shown that Johnstone was misled by any industry group.

Some of the authors’ scholarship is quite careless.  For instance, they assert that

“By the mid 1940s, the international scientific community had recognized the link between asbestos and cancer.10–18”

Readers should review all the endnotes, 10 – 18, but endnote 12 is especially interesting:

“12 Macklin MT, Macklin CC. Does chronic irritation cause primary carcinoma of the human lung? Arch Path. 1940;30:924–55.”

As I have noted before, the Macklins, and especially Dr. Madge Macklin, brought a great deal of rigor and skepticism to broad claims about the causation of lung cancer. See “Silicosis, Lung Cancer, and Evidence-Based Medicine in North America” (July 4, 2014).  This citation and others do not appear to support the sweep of Egilman and his student authors’ claim.

The next mention of silica occurs in the context of an allegation that corporations (presumably not plaintiffs’ laywers) have worked to “disguise” health concerns and influence governmental policy about several products, materials, including silica:

“During the last several decades, researchers in a wide spectrum of fields have documented the direct and purposeful efforts of corporations to disguise public health concerns and affect government policies, particularly in the tobacco, alcohol, silica, and asbestos industries, and more recently, the pharmaceutical, chemical, and ultra-processed food and drink industries.79,73

Egilman at 121.

The authors’ citations, however, do not support any such allegation about silica. Endnote 73[3] is an article by Egilman, and others, on Vioxx; and endnote 79[4] is an article about alcohol, tobacco, and foods. In the very next sentence, the authors further claim that:

“Corporate-funded ‘objective science’ leading to the corruption of scientific literature remains a major problem.65,68,69,71,73,75,80–86

Once again, none of the endnotes (65, 68, 69, 71, 75, and 80-86) supports the authors’ claim that anyone in the mining, milling, or marketing of crystalline silica has funded science in a way that led to the corruption of the scientific literature. Not surprisingly, the authors ignore the frauds perpetuated by litigation industry players. See, e.g., In re Silica Products Liab. Lit., 398 F. Supp. 2d 563 (S.D. Tex. 2005) (federal trial judge rebukes the litigation industry for fraudulent claiming in MDL 1553).


[1] The article acknowledges that Ms. Bird and Ms. Lee were employees of Dr. Egilman.  Ms. Bird appears now to be a student in the U.K., studying medical anthropology.  Ms. Bird, and Ms. Lee, appeared on earlier works by Egilman.  See, e.g., David S Egilman, Tess Bird, and Caroline Lee, “MetLife and its corporate allies: dust diseases and the manipulation of science,” 19 Internat’l J. Occup. & Envt’l Health 287 (2013); David Steven Egilman, Emily Laura Ardolino, Samantha Howe, and Tess Bird, “Deconstructing a state-of-the-art review of the asbestos brake industry,” 21 New Solutions 545 (2011).

[2] Ms. Lee appears to have been employed for Egilman’s litigation consulting firm, Never Again Consulting, from 2011 until August 2013, when she entered the University of Maryland law school.

[3] Krumholz HM, Ross JS, Presler AH, Egilman DS. What have we learnt from Vioxx. Br Med J. 2007;334(7585):120–3.

[4] Moodie R, Stuckler D, Montiero C, Sheron N, Neal B, Thamarangsi T, et al. Profits and pandemics: prevention of harmful effects of tobacco, alcohol, and ultra-processed food and drink industries. Lancet. 2013;381:670–79.

Discovery of Litigation Financing – The Jackpot Justice Finance Corporation

July 16th, 2014

Over two years ago, I wrote that courts and counsel have not done enough to adapt to the litigation industry’s use of third-party financing. SeeLitigation Funding” (May 8, 2012). A few days ago, Byron Stier, at the Mass Tort Litigation Blog, posted a short news item about a recent effort to modify discovery rules to take into account the litigation industry’s business model of seeking third-party litigation funding.  SeeIndustry Groups Seek Amendment of Rule 26 to Require Disclosure of Third Party Litigation Financing” (July 13, 2014).

Stier reported that the U.S. Chamber of Commerce Institute for Legal Reform, American Insurance Association, American Tort Reform Association, Lawyers for Civil Justice, and National Association of Manufacturers, back in April 2014, wrote a letter to the Committee on Rules of Practice and Procedure of the Administrative Office of the federal courts, to propose an amendment to Federal Rule of Civil Procedure 26(a)(1)(A). Their proposed new language is underscored, and follows 26(a)(1)(A)(i)-(iv):

“(v) for inspection and copying as under Rule 34, any agreement under which any person, other than an attorney permitted to charge a contingent fee representing a party, has a right to receive compensation that is contingent on, and sourced from, any proceeds of the civil action, by settlement, judgment or otherwise.”

This proposal is important and necessary to ensure that defendants can inquire about financial bias in jury voir dire, as well as identify bias among judges, special masters, and witnesses.  Other procedural rule reforms will be needed as well. Appellate briefs should disclose financiers that have a stake in the litigation. When courts limit counsel and parties’ communications to the media about a case, such limits should apply as well to insurers and third-party financiers of the litigation efforts. Third-party financiers provide a convenient way for the litigation industry to lobby regulators and legislators, and more expansive disclosure rules are needed to capture the activities of the litigation financiers.  Funding of litigation-related research by third-party financiers should be anticipated by journal editors with more expansive disclosure rules; journal editors should be alert to evolving financial markets that may influence research agendas and publications. Lawyers’ scrutiny of new clients for conflicts now requires inquiry into the veiled interests created by litigation financing.

Too Many Narratives – Historians in the Dock

July 13th, 2014

Historical Associates Inc. (HAI) is a commercial vendor for historical services, including litigation services. Understandably, this firm, like the academic historians who service the litigation industry, takes a broad view of the desirability of historian expert witness testimony.  An article in one of the HAI’s newsletters stakes out lawyer strategies in trying to prove historical facts.  Lawyers can present percipient witnesses, or they

“can present the story themselves, but in the end, arguments by advocates can raise questions of bias that obscure, rather than clarify, the historical facts at issue.”

Mike Reis and Dave Wiseman, “Introducing and interpreting facts-in-evidence: the historian’s role as expert witness,” HAIpoints 1 (Summer 2010)[1]. These commercial historians recommend that advocacy bias, so clear in lawyers’ narratives, be diffused or obscured by having a professional historian present the “story.”  They tout the research skills of historians: “Historians know how to find critical historical information.” And to be sure, historians, whether academic or for-hire may offer important bibliographic services, as well as help in translating, authenticating, and contextualizing documents.  But these historians from HAI want a role on center-stage, or at least in the witness box.  They tell us that:

“Historians synthesize information into well-documented, compelling stories.”

Ah yes, compelling stories, as in “the guiltless gust of a rattling good yarn[2].” The legal system should take a pass on such stories.

*     *     *     *     *     *

A recent law review article attempts to provide a less commercial defense of expert witness testimony.  See Alvaro Hasani, “Putting history on the stand: a closer look at the legitimacy of criticisms levied against historians who testify as expert witnesses,” 34 Whittier L. Rev. 343 (2013) [Hasani].  Hasani argues that historians strive to provide objective historical “interpretation,” by selecting reliable sources, and reliably reading and interpreting these sources to create a reliable “narrative.” Hasani at 355. Hasani points to some courts that have thrown up their hands and declared Daubert reliability factors inapplicable to non-scientific historian testimony. See, e.g., United States v. Paracha, No. 03 CR. 1 197(SHS), 2006 WL 12768, at *19 (S.D.N.Y. Jan. 3, 2006) (noting that Daubert is not designed for gatekeeping of a non-scientific, historian expert witness’s methodology); Saginaw Chippewa Indian Tribe of Michigan v. Granholm, 690 F. Supp. 2d 622, 634 (E.D. Mich. 2010) (noting that “[t]here is no way to ‘test’ whether the experts’ testimony concerning the historical understanding of the treaties is correct. Nor is it possible to establish an ‘error rate’ for historical experts.”).

Not all testifying historians agree, however, that their research and findings are non-scientific.  Here is how one plaintiffs’ expert witness characterized historical thinking:

“Q. Do you believe that historical thinking is a form of scientific thinking?

A. I do. I think that history is sometimes classed with the humanities, sometimes classed with the social sciences, but I think there is a good deal of historical research and writing that is a form of social science.”

Examination Before Trial of Gerald Markowitz, in Mendez v. American Optical, District Court for Tarrant County, Texas (342d Judicial District), at 44:13-20 (July 19, 2005). Professor Susan Haack, and others, have made a persuasive case that the epistemic warrants for claims of knowledge, whether denominated scientific or non-scientific, are not different in kind. If historian testimony is not about knowledge of the past, then it clearly has no role in a trial. Furthermore, Professor Markowitz is correct that sometimes historical opinions are scientific in the sense that they can be tested. If a labor historian asserts that workers are exploited and subjected to unsafe work conditions due to the very nature of capitalism and the profit motives, then that historian’s opinion will be substantially embarrassed by the widespread occupational disease in European and Asian communist regimes.

When Deborah Lipstadt described historian David Irving as a holocaust denier[3], Irving sued Lipstadt for defamation.  In defending against the claim, Lipstadt successfully carried the burden of proving the truth of her accusation.  The trial court’s judgment, quoted by Hasani, reads like a so-called Daubert exclusion of plaintiff Irving’s putative historical writing. Irving v. Penguin Books Ltd., No. 1996-1-1113, 2000 WL 362478, at ¶¶ 1.1, 13.140 (Q.B. Apr. 11, 2000)(finding that “Irving ha[d] misstated historical evidence; adopted positions which run counter to the weight of the evidence; given credence to unreliable evidence and disregarded or dismissed credible evidence.”).

The need for gatekeeping of historian testimony should be obvious.  Historian testimony is often narrative of historical fact that is not beyond the ken of an ordinary fact finder, once the predicate facts are placed into evidence.  Such narratives of historical fact present a serious threat to the integrity of fact finding by creating the conditions for delegation and deferring fact finding responsibility to the historian witness, with an abdication of responsibility by the fact finder. See Ronald J. Allen, “The Conceptual Challenge of Expert Evidence,” 14 Discusiones Filosóficas 41, 50-53 (2013).

Some historians clearly believe that they are empowered by the witness chair to preach or advocate. Allan M. Brandt, who has served as a party expert witness to give testimony on many occasions for plaintiffs in tobacco cases, unapologetically described the liberties he has taken thus:

“It seems to me now, after the hopes and disappointments of the courtroom battle, that we have a role to play in determining the future of the tobacco pandemic. If we occasionally cross the boundary between analysis and advocacy, so be it. The stakes are high, and there is much work yet to do.”

Allan M. Brandt, The Cigarette Century: The Rise, Fall, and Deadly Persistance of the Product That Defined American 505 (2007).

Hasani never comes to grips with the delegation problem or with Brandt’s attitude, which is quite prevalent in the product liability arena. The problem is more than merely “occasional.” The overreaching by historian witnesses reflects the nature of their discipline, the lack of necessity for their testimony, and the failure of courts to exercise their gatekeeping. The problem with Brandt’s excuse making is that neither analysis nor advocacy is needed or desired. Advocacy is the responsibility of counsel, as well as the kind of analysis involved in much of historian testimony.  For instance, when historians offer testimony about the so-called “state of the art,” they are drawing inferences from published and unpublished sources about what people knew or should have known, and about their motivations.  Although their bibliographic and historical researches can be helpful to the fact finder’s effort to understand who was writing what about the issue in times past, historians have no real expertise, beyond the lay fact finder, in discerning intentions, motivations, and belief states.

Hasani concludes that the prevalence of historian expert witness testimony is growing. Hasani at 364.  He cites, however, only four cases for the proposition, three of which pre-date Daubert.  The fourth is an native American rights case. Hasani at 364 n.139. There is little or no evidence that historian expert witness testimony is becoming more prevalent, although it continues in product liability where state of the art — who knew what, when — remains an issue in strict liability and negligence. Mack v. Stryker Corp., 893 F. Supp. 2d 976 (D. Minn. 2012), aff’d, 748 F.3d 845 (8th Cir. 2014). There remains a need for judicial vigilance in policing such state-of-the-art testimony.


[1] Mike Reis is the Vice President and Director of Litigation Research at History Associates Inc. Mr. Reis was received his bachelor’s degree from Loyola College, and his master’s degree from George Washington University, both in history. David Wiseman, an erstwhile trial attorney, conducts historical research for History Associates.

[2] Attributed to Anthony Burgess.

[3] Deborah E. Lipstadt, Denying the Holocaust: The Growing Assault on Truth and Memory 8 (1993).

 

The opinions, statements, and asseverations expressed on Tortini are my own, or those of invited guests, and these writings do not necessarily represent the views of clients, friends, or family, even when supported by good and sufficient reason.