Schachtman Law

TORTINI

For your delectation and delight, desultory dicta on the law of delicts.

Bad and Good Statistical Advice from the New England Journal of Medicine

July 2nd, 2011

Many people consider The New England Journal of Medicine (NEJM) a prestigious journal. It is certainly widely read. Judging from its “impact factor,” we know the journal is frequently cited. So when the NEJM weighs in on issue that involves the intersection of law and science, I pay attention.

Unfortunately, this week’s issue contains an editorial “Perspective” piece that is filled with incoherent, inconsistent, and incorrect assertions, both on the law and the science. Mark A. Pfeffer and Marianne Bowler, “Access to Safety Data – Stockholders versus Prescribers,” 364 New Engl. J. Med. ___ (2011).

Dr. Mark Pfeffer and the Hon. Marianne Bowler used the recent United States Supreme Court decision in Matrixx Initiatives, Inc. v. Siracusano, __ U.S. __, 131 S.Ct., 1309 (2011), to advance views, not supported by the law or the science. Remarkably, Dr. Pfeffer is the Victor J. Dzau Professor of Medicine, at the Harvard Medical School. He is both a physician, and he has received a Ph.D. degree in physiology and biophysics. Ms. Bowler is both a lawyer and a federal judge. Between the two, they should have provided better, more accurate, and more consistent advice.

1. The Authors Erroneously Characterize Statistical Significance in Inappropriate Bayesian Terms

The article begins with a relatively straightforward characterization of various legal burdens of proof. The authors then try to collapse one of those burdens of proof, “beyond a reasonable doubt,” which has no accepted quantitative meaning, to a significance probability that is used to reject a pre-specified null hypothesis in scientific studies:

“To reject the null hypothesis (that a result occurred by chance) and deem an intervention effective in a clinical trial, the level of proof analogous to law’s ‘beyond a reasonable doubt’ standard would require an extremely stringent alpha level to permit researchers to claim a statistically significant effect, with the offsetting risk that a truly effective intervention would sometimes be deemed ineffective. Instead, most randomized clinical trials are designed to achieve a lower level of evidence that in legal jargon might be called ‘clear and convincing’, making conclusions drawn from it highly probable or reasonably certain.”

Now this is both scientific and legal nonsense. It is distressing that a federal judge characterizes the burden of proof that she must apply, or direct juries to apply, as “legal jargon.” More important, these authors, scientist and judge, give questionable quantitative meanings to burdens of proof, and they misstate the meaning of statistical significance. When judges or juries must determine guilt “beyond a reasonable doubt,” they are assessing the prosecution’s claim that the defendant is guilty, given the evidence at trial. This posterior probability can be represented as:

Probability (Guilt | Evidence Adduced)

This is what is known as a posterior probability, and it is fundamentally different from significance probability.

The significance probability is a transposed conditional probability from the posterior probability that is used to assess guilt in a criminal trial, or contentions in a civil trial. As law professor David Kaye and his statistician coauthor, the late David Freedman, described the p-value and significance probability:

“The p-value is the probability of getting data as extreme as, or more extreme than, the actual data, given that the null hypothesis is true:

p = Probability (extreme data | null hypothesis in model)

* * *

Conversely, large p-values indicate that the data are compatible with the null hypothesis: the observed difference is easy to explain by chance. In this context, small p-values argue for the plaintiffs, while large p-values argue for the defense.¹³¹Since p is calculated by assuming that the null hypothesis is correct (no real difference in pass rates), the p-value cannot give the chance that this hypothesis is true. The p-value merely gives the chance of getting evidence against the null hypothesis as strong or stronger than the evidence at hand—assuming the null hypothesis to be correct. No matter how many samples are obtained, the null hypothesis is either always right or always wrong. Chance affects the data, not the hypothesis. With the frequency interpretation of chance, there is no meaningful way to assign a numerical probability to the null hypothesis.¹³²”

David H. Kaye and David A. Freedman, “Reference Guide on Statistics,” Federal Judicial Center, Reference Manual on Scientific Evidence 122 (2ed. 2000). Kaye and Freedman explained over a decade ago, for the benefit of federal judges:

“As noted above, it is easy to mistake the p-value for the probability that there is no difference. Likewise, if results are signiﬁcant at the .05 level, it is tempting to conclude that the null hypothesis has only a 5% chance of being correct.¹⁴²

This temptation should be resisted. From the frequentist perspective, statistical hypotheses are either true or false; probabilities govern the samples, not the models and hypotheses. The signiﬁcance level tells us what is likely to happen when the null hypothesis is correct; it cannot tell us the probability that the hypothesis is true. Signiﬁcance comes no closer to expressing the probability that the null hypothesis is true than does the underlying p-value.¹⁴³”

Id. at 124-25.

As we can see, our scientist from the Harvard School of Medical School and our federal judge have committed the transpositional fallacy by likening “beyond a reasonable doubt” to the alpha used to test for a statistically significant outcome in a clinical trial. They are not the same; nor are they analogous.

This fallacy has been repeatedly described. Not only has the Reference Manual on Scientific Manual (which is written specifically for federal judges) described the fallacy in detail, but legal and scientific writers have urged care to avoid this basic mistake in probabilistic reasoning. Here is a recent admonition from one of the leading writers on the use (and misuse) of statistics in legal procedures:

“Some commentators, however, would go much further; they argue that is an arbitrary statistical convention and since preponderance of the evidence means 51% probability, lawyers should not use 5% as the level of statistical significance but 49% – thus rejecting the null hypothesis when there is up to a 49% chance that it is true. In their view, to use a 5% standard of significance would impermissibly raise the preponderance of evidence standard in civil trials. Of course the 5% figure is arbitrary (although widely accepted in statistics) but the argument is fallacious. It assumes that 5% (or 49% for that matter) is the probability that the null hypothesis is true. The 5% level of significance is not that, but the probability of the sample evidence if the null hypothesis were true. This is a very different matter. As I pointed out in Chapter1, the probability of the sample given the null hypothesis is not generally the same as the probability of the null hypothesis given the sample. To relate the level of significance to the probability of the null hypothesis would require an application of Bayes’s theorem and the assumption of a prior probability distribution. However, the courts have usually accepted the statistical standard, although with some justifiable reservations when the P-value is only slightly above the 5% cutoff.”

Michael O. Finkelstein, Basic Concepts of Probability and Statistics in the Law 54 (N.Y. 2009) (emphasis added).

2. The Authors, Having Mischaracterized Burden-of-Proof and Significance Probabilities, Incorrectly Assess the Meaning of the Supreme Court’s Decision in Matrixx Initiatives.

I have written a good bit about the Court’s decision in Matrixx Initiatives, most recently with David Venderbush, for the Washington Legal Foundation. See Schachtman & Venderbush, “Matrixx Unbounded: High Court’s Ruling Needlessly Complicates Scientific Evidence Principles,” W.L.F. Legal Backgrounder (June 17, 2011).

I was thus startled to see the claim of a federal judge that the Supreme Court, in Matrixx, had “applied the ‘fair preponderance of the evidence’ standard of proof used for civil matters.” Matrixx was a case about the sufficiency of the pleadings, and thus there really could have been no such application of a burden of proof to an evidentiary display. The very claim is incoherent, and at odds with the Supreme Court’s holding.

The NEJM authors went on to detail how the defendant in Matrixx had persuaded the trial court that the evidence against its product, Zicam, did not reach statistical significance, and therefore the evidence should not be considered “material.” As I have pointed out before, Matrixx focused on adverse event reports, as raw number of reported events, which did not, and could not, be analyzed for statistical significance. The very essence of Matrixx’s argument was nonsense, which perhaps explains the company’s nine-nothing loss in the Supreme Court. The authors of the opinion piece in the NEJM, however, missed that it is not the evidence of adverse event reports, with or without a statistical analysis, that is material. What was at issue was whether the company’s failure to disclose this information, along with a good deal more information, in the face of the company’s having made very aggressive, optimistic sales and profits projections for the future.

The NEJM authors proceed to tell us, correctly, that adverse events do not prove causality, but then they tell us, incorrectly, that the Matrixx case shows that “such a high level of proof did not have to be achieved.” While the authors are correct about the sufficiency of adverse event reports for causal assessments, they miss the legal significance of there being no burden of proof at play in Matrixx; it was a case on the pleadings. The issue was the sufficiency of those pleadings, and what the Supreme Court made clear was that in the context of a product subject to FDA regulation, causation was never the test for materiality because the FDA could withdraw the product on a showing far less than scientific causation of harm. So the plaintiffs could allege less than causation, and still have pleaded a sufficient case of securities fraud. The Supreme Court did not, and could not, address the issue that the NEJM authors discuss. The authors’ assessment that the Matrixx case freed legal causation of any requirement of statistical significance is a tortured reading of obiter dictum, not the holding of the case. This editorializing is troubling.

The NEJM authors similarly hold forth on what clinicians consider material, and they announce that “[c]linicians are well aware that to be considered material, information regarding drug safety does not have to reach the same level of certainty that we demand for demonstrating efficacy.” This is true, but clinicians are ethically bound to err on the side of safety: Primum non nocere. See, e.g., Tamraz v. Lincoln Elec. Co., 620 F.3d 665, 673 (6^th Cir. 2010) (noting that treating physicians have more training in diagnosis than in etiologic assessments), cert. denied, ___ U.S.____ (2011). Again, the authors’ statements have nothing to do with the Matrixx case, or with the standards for legal or scientific causation.

3. The Authors, Inconsistently with Their Characterization of Various Probabilities, Proceed Correctly To Describe Statistical Significance Testing for Adverse Outcomes in Trials.

Having incorrectly described beyond a reasonable doubt as like p <0.05, the NEJM authors then, correctly point out that standard statistical testing cannot be used for “evaluating unplanned and uncommon adverse events.” The authors also note that the flood of data in the assessment of causation of adverse events is filled with “biologic noise.” Physicians and regulators may take the noise signals and claim that they hear a concert. This is exactly why we should not confuse precautionary judgments with scientific assessments of causation.

Posted in Rule 702, statistical evidence | 1 Comment »

Ninth Circuit Affirms Rule 702 Exclusion of Dr David Egilman in Diacetyl Case

June 20th, 2011

On June 17, 2011, the Ninth Circuit of the United States Court of Appeals affirmed a district judge’s decision to exclude Dr David S. Egilman from testifying in a consumer-exposure diacetyl case. Newkirk v. Conagra Foods Inc. (9^th Cir. 2011).

Plaintiff claimed to develop bronchiolitis obliterans from having popped and eaten an Homeric quantity of microwavable popcorn. The case was thus a key test of “consumer” diacetyl exposure. Another case, also involving Egilman, just finished a Daubert hearing in Colorado, last week.

To get the full “flavor” of this diacetyl case, you may have to read the district court’s opinion, which excluded Egilman and other witnesses, and entered summary judgment for the defense. Newkirk v. Conagra Foods, Inc., No. CV-08-273, 2010 WL 2680184 (E.D. Wash. July 2, 2010).

Plaintiff appealed, and so did Egilman. (See attached Egilman Motion Appeal Diacetyl Exclusion 2011 and Egilman Declaration Newkirk Diacetyl Appeal 2011.) In what some may consider scurrilous pleading, Egilman attacked the district judge for having excluded him from testifying. If Egilman’s challenge to the trial judge was not bizarre enough, Egilman also claimed a right to intervene in the appeal by advancing the claim that the Rule 702 exclusion hurt his livelihood. The following language is from paragraph 11 of Dr. Egilman’s declaration in support of his motion:

“The Daubert ruling eliminates my ability to testify in this case and in others. I will lose the opportunity to bill for services in this case and in others (although I generally donate most fees related to courtroom testimony to charitable organizations, the lack of opportunity to do so is an injury to me). Based on my experience, it is virtually certain that some lawyers will choose not to attempt to retain me as a result of this ruling. Some lawyers will be dissuaded from retaining my services because the ruling is replete with unsubstantiated pejorative attacks on my qualifications as a scientist and expert. The judge’s rejection of my opinion is primarily an ad hominem attack and not based on an actual analysis of what I said – in an effort to deflect the ad hominem nature of the attack the judge creates ‘straw man’ arguments and then knocks the straw men down, without ever addressing the substance of my positions.”

Egilman Declaration at Paragraph 11.

Egilman tempers his opinion about the prejudice he will suffer in front of judges in future cases. Only judges who have not seen him before would likely be persuaded by Judge Peterson’s decision in Newkirk. Those judges who have heard him testify before would, no doubt, see him for the brilliant crusading avenger that he is:

“This will generally not occur in cases heard before Judges where I have already appeared as a witness. For example a New York state trial judge has praised plaintiffs’ molecular-biology and public-health expert Dr. David Egilman as follows: ‘Dr. Egilman is a brilliant fellow and I always enjoy seeing him and I enjoy listening to his testimony . . . . He is brilliant, he really is.’ [Lopez v. Ford Motor Co., et al. (120954/2000; In re New York City Asbestos Litigation, Index No. 40000/88).]”

Egilman Declaration at p. 9 n. 2.

It does not appear as though Egilman’s attempt to intervene helped plaintiff before the Ninth Circuit, which may not have thought that he was as brilliant as the unidentified trial judge in Lopez.

The Newkirk case is interesting for several reasons.

First, the Circuit correctly saw that general causation must be shown before the plaintiff can invoke a differential etiology analysis.

Second, the Circuit saw that it is not sufficient that the substance in question can cause the outcome claimed; the substance must do so at the levels of exposure that were experienced by the plaintiff. In Newkirk, even by consuming massive quantities of microwave popcorn, plaintiff had not shown exposure levels to diacetyl equivalent to the exposures among factory workers at risk for bronchiolitis obliterans. The affirmance of the district court is a strong statement that exposure matters in the context of the current understanding of diacetyl causation.

Third, the Circuit was not intimidated or persuaded by the tactics of Dr David Egilman, expert witness.

Fourth, having dealt with the issues deftly, the Ninth Circuit issued a judgment from which there will be no appeal.

Posted in Rule 702 | 3 Comments »

WLF Legal Backgrounder on Matrixx Initiatives

June 20th, 2011

In Matrixx Initiatives, Inc. v. Siracusano, ___ U.S. ___, ___ , 2011 WL 977060 (Mar. 22, 2011), the Supreme Court addressed a securities fraud case against an over-the-counter pharmaceutical company for speaking to the market about its rosy financial projections, but failing to provide information received about the hazards of the product.

Much or most of the holding of the case is an unexceptional application of settled principles of securities fraud litigation in the context of claims against a pharmaceutical company with products liability cases pending. The defendant company, however, attempted to import Rule 702 principles of scientific evidence into a motion to dismiss on the pleadings, with much confusion resulting among the litigants, the amici, and the Court. The Supreme Court ruled unanimously to affirm the reinstatement of the complaint against the defendant.

I have written about this case previously: “The Matrixx – A Comedy of Errors,” and “Matrixx Unloaded,” and “The Matrixx Oversold,” and “De-Zincing the Matrixx.”

Now, with the collaboration of David Venderbush from Alston & Bird LLP, we have collected our thoughts to share in the form of a Washington Legal Foundation Legal Backgrounder, which is available for download at the WLF’s website. Schachtman & Venderbush, “Matrixx Unbounded: High Court’s Ruling Needlessly Complicates Scientific Evidence Principles,” 26 (14) Legal Backgrounder (June 17, 2011).

Posted in Rule 702 | No Comments »

The Fascinating Thing About History

June 16th, 2011

“There is something fascinating about science. One gets such wholesale returns of conjecture out of such a trifling investment of fact.”

— Mark Twain, Life on the Mississippi 1883.

History is much more fun and profitable, because you can get in on the action, without any investment of fact. The SEC ought to look into this.

Consider the historians of silicosis:

“In the postwar era, professionals, industry, government, and a conservative labor movement tried to bury silicosis as an issue.”

David Rosner & Gerald Markowitz, Deadly Dust: Silicosis and the Politics of Occupational Disease in the Twentieth Century America 213 (Princeton 1991).

Now, I pick on Rosner and Markowitz because they pick on me, they are easy targets, and their writings are illustrative of what I believe is so wrong about importing historians’ broad, and sometimes glib, judgments into the courtroom. Of course, Rosner and Markowitz have made themselves actors in various litigation efforts to advance their radical and Marxist views. See Ronald Johnston & Arthur McIvor, “Workshop Handout: Approaches and Methods in the History of Occupational and Environment Health,” presented at The Fourth Annual International Conference on the History of Occupational and Environmental Health (June 2010)(asking whether Rosner and Markowitz are not the best writing in the tradition of radical and Marxist approaches to the history of workers’ health).

Before World War II, there were notable, unfortunate large scale outbreaks of silicosis. The silicosis outbreaks among workers in the Barre, Vermont, granite quarries and sheds, and of course, among workers at the Gauley Bridge/Hawk’s Nest tunnel, are among the most notorious.

After WWII, the most notable outbreak was a fantasy and a fraud, created by plaintiffs’ counsel who conspired with reprobate physicians. In re Silica Products Liab. Litig., 398 F.Supp. 2d 563 (S.D.Tex. 2005) (Jack, J.).

Occasionally, empirical evidence can be brought to bear to embarrass the glib generalizations that historians make. Consider the claim, above, by Rosner & Markowitz that everyone (other than the heroic Leninist labor unions) sought to bury the issue of silicosis.

Well, we can obtain something of a reality check by measuring the number of publications in the National Library of Medicine’s database (PubMed) on silicosis. A simply search on “silicosis,” with limits to each decade after the 1930s reveals a pattern that silicosis had not been buried at all:

Date Range Number of Articles from Keyword Search

1940 – 1949 113

1950 – 1959 1,421

1960 – 1969 1,867

1970 – 1979 1,178

1980 – 1989 940

1990 – 1999 882

2000 – 2009 843

Considering the major post-War developments in the medical world, from antibiotics, poliomyelitis, tobacco-related cancers, and other chronic diseases, the continued interest in silicosis after World War II is remarkable.

I suppose that Rosner and Markowitz would discount the words of the Centers for Disease Control (CDC), because, after all, the CDC is part of government, which in turn is part of the conspiracy with medical professionals, industry, and non-Marxist labor union leaders, to bury the silicosis issues. If you see through Rosner and Markowitz’ broad brush generalization, you might be interested to know that the CDC mentioned silicosis as among the ten great public achievements of the twentieth century. CDC, “Ten Great Public Health Achievements – United States, 1900 – 1999,” 48(12) CDC Morbidity and Mortality Weekly Report 241 (April 02, 1999)(“Work-related health problems, such as coal workers’ pneumoconiosis (black lung), and silicosis — common at the beginning of the century — have come under better control.”).

This brand of historical generalizations does not belong in courtrooms.

Posted in Historian Testimony | No Comments »

Conflicts of Interest, Footnote 17, and Scientific McCarthyism

June 12th, 2011

In Exxon Shipping Co. v. Baker, 554 U.S. 471, 501 (2008) – the Exxon Valdez case – the Supreme Court struck down a $2.5 billion punitive damage award. Justice Souter wrote the opinion for a fragmented court, in a 5-3 decision, with several concurrences. There are many interesting aspects to the case, including a curious statement with a more curious footnote. Justice Souter wrote:

“We are aware of no scholarly work pointing to consistency across punitive awards in cases involving similar claims and circumstances.¹⁷“

In the corresponding footnote, Justice Souter explained:

“The Court is aware of a body of literature running parallel to anecdotal reports, examining the predictability of punitive awards by conducting numerous “mock juries,” where different “jurors” are confronted with the same hypothetical case. See, e.g., C. Sunstein, R. Hastie, J. Payne, D. Schkade, W. Viscusi, Punitive Damages: How Juries Decide (2002); Schkade, Sunstein, & Kahneman, Deliberating About Dollars: The Severity Shift, 100 Colum. L.Rev. 1139 (2000); Hastie, Schkade, & Payne, Juror Judgments in Civil Cases: Effects of Plaintiff’s Requests and Plaintiff’s Identity on Punitive Damage Awards, 23 Law & Hum. Behav. 445 (1999); Sunstein, Kahneman, & Schkade, Assessing Punitive Damages (with Notes on Cognition and Valuation in Law), 107 Yale L.J. 2071 (1998). Because this research was funded in part by Exxon, we decline to rely on it.”

Professor Sunstein, then at University of Chicago, of course now serves in President Obama’s administration. Professor Kahneman is a Nobel Laureate. Professor Viscusi has been one of the most prolific writers about and investigators of punitive damages. Justice Souter’s footnote could easily be interpreted to impugn the integrity of their research by virtue of their corporate sponsorship.

The footnote was curious in large part because Exxon won the case, which leaves open why Justice Souter went out of his way to call into question research that supported his concern about the vagaries of punitive damage awards. Having in large measure adopted an approach urged by the Exxon-sponsored studies, Justice Souter’s disclaimer seems rather disingenuous. Perhaps Justice Souter was simply acknowledging that the Court was aware of the work, and for sake of appearances, wanted to note that the Court had reached its decision independently of the litigant-sponsored studies.

There was nothing underhanded done by Exxon; the studies disclosed their funding source.

Since the Exxon Valdez case, expert witnesses, litigants, and courts have pointed to footnote17 inappropriately to suggest that party-sponsored studies should be disregarded without consideration of their merits.

For almost a century, litigants have invoked social science research designed to influence court’s decisions about “legislative facts” or policy. Such research is very different from research studies upon which expert witnesses rely when appearing before trial courts and juries, responsible for finding facts.

Footnote 17 was thus torn from its context of using social science research to shape policy, and extended to apply to scientific studies that are relied upon by expert witnesses at the trial court level. Overlooking this distinction, Judge Jack Weinstein jumped on the issue when he commented on sponsorship of pharmaceutical companies’ clinical trials, and generalized (without any reliable scientific basis) that the “commercial bias found in today’s research laboratories means studies are often lacking in essential objectivity, with the potential for misinformation, skewed results or cover-ups….” In re Zyprexa Prod. Liab. Litig., 253 F.R.D. 69, 106-08 (E.D.N.Y. 2008) (citing Exxon).

A new front of “Scientific McCarthyism” has opened. This intolerance toward corporate sponsorship has been going on for some time. Journal editors and industry critics have been using the “conflicts of interest” mantra to impugn industry-sponsored studies, and to impose greater burdens on the publication of such studies than required for federally funded studies. Curiously, the same journal editors have stuck their heads in the sand when it comes to studies sponsored by plaintiffs’ counsel, or conducted by scientists who are consultants to, or witnesses for, plaintiffs’ counsel in tort cases.

Back in 1993, Ken Rothman referred to this anti-industry as the “new McCarthyism in science.” Kenneth J. Rothman, “Conflict of interest: the new McCarthyism in science,” 269 J. Am. Med. Ass’n 2782 (1993). The McCarthyites were undeterred. The anti-industry journals pushed forward by increasing the burdens on industry sponsored studies, especially on pharmaceutical clinical trials. See, e.g., Catherine D. DeAngelis, P. B. Fontanarosa, and A. Flanagin, “Reporting financial conflicts of interest and relationships between investigators and research sponsors 286 J. Am. Med. Ass’n 89 (2001)

Courageously, some scientists fought for science to be judged on its merits. See Thomas P. Stossel, “Has the hunt for conflicts of interest gone too far?” 336 Brit. Med. J. 476 (2008); Nature Publishing Group, “Nothing to see here: based on one company’s past poor publishing practices, a top-tier medical journal misguidedly stigmatizes any paper from industry,” 26 Nature Biotechnol. 476 (2008); Kenneth J. Rothman & S. Evans, “Extra scrutiny for industry funded trials: JAMA’s demand for an additional hurdle is unfair–and absurd, 331 Brit. Med. J. 1350 (2005), and 332 Brit. Med. J. 151 (2006) (erratum).

Professor Stossel and others created an organization, ACRE – The Association of Clinical Researchers and Educators, to defend legitimate interactions between Physicians and Industry. ACRE has spoken out against the lopsided demonization of the pharmaceutical industry, and the lionizing of the industry’s critics.

The anti-industry prejudice seemed to jump the shark when a gaggle of plaintiffs’ expert witnesses published a “case study” of publication abuses allegedly perpetrated by Merck. Some of the authors were paid expert witnesses in litigation against Merck, but this unseemly conflict of interest did not seem to disturb the journal’s editors. J. S. Ross, K. P. Hill, David S. Egilman, and Harlan M. Krumholz, “Guest authorship and ghostwriting in publications related to rofecoxib: a case study of industry documents from rofecoxib litigation. 299 J. Am. Med. Ass’n 1800 (2008); Catherine D. DeAngelis & P.B. Fontanarosa, “Impugning the integrity of medical science: the adverse effects of industry influence,” 299 J. Am. Med. Ass’n 1833 (2008).

Along with Ken Rothman, another bold voice has cried out against the unfairness and partiality of journals’ conflict-of-interest rules and policies. Laurence J. Hirsch, “Conflicts of Interest, Authorship, and Disclosures in Industry-Related Scientific Publications: The Tort Bar and Editorial Oversight of Medical Journals,” 84 Mayo Clin. Proc. 811 (2009). Dr. Hirsch published a strongly worded commentary that journals’ concerns are often poorly disguised prejudices against industry. Many of the journals in question rarely or never fuss over the obvious conflicts of interest created by the “profit motive” of researchers who want to climb the academic ladder, increase their salaries, enlarge their budgets, extend their influence, travel to organizational conferences, bolster their prestige, win more grants, enhance their reputations, or advance their political goals or ideologies.

Dr Hirsch uses the publication by Ross, Hill, Egilman, and Krumholz as an example of the double standard. Their publication in the Journal of the American Medical Association was accompanied by an anemic disclosure that they had been consultants to the plaintiffs in Vioxx litigation, but they neglected to mention that they had actually testified for plaintiffs, and had earned thousands of dollars for their efforts.

Dr Hirsch published a correction last year in which he noted that “Dr Egilman has not testified in court in breast implant and connective tissue disease, or in antidepressant or antipsychotic drug cases.” Laurence J. Hirsch, “Corrections,” 85 Mayo Clin. Proc. 99 (2010). This correction was curious because Dr Egilman had testified in a breast implant case: Vasallo v. Baxter Healthcare, tried in Massachusetts, in the late 1990s. The Vassallo case involved allegations that silicone had caused systemic disease, an allegation that was ultimately shown to be meritless.

Judge Jack B. Weinstein, “Preliminary Reflections on Administration of Complex Litigation.” Cardozo Law Review DeNovo 1, 14 (2009) (“[t]he breast implant litigation was largely based on a litigation fraud. … Claims—supported by medical charlatans—that enormous damages to women’s systems resulted could not be supported.”)

The medical profession, the courts, and the public are seriously misled by the obsession with conflicts of interest, on either side. The obsession allows a disclosed or undisclosed conflict of interest to substitute for the much harder work of considering the merits of a study.

Posted in Conflicts of Interest | No Comments »

National Academies Press Publications Are Now Free

June 3rd, 2011

Publications of the National Research Council, as well as those of its constitutive organizations, the National Academy of Science, the Institute of Medicine, and the National Academy of Engineering, are often important resources for lawyers who litigate scientific and technical issues. Right or wrong, these publications become forces in their own right in the courtroom, where they command serious attention from trial and appellate judges.

According to the National Academies Press’s website, all electronic versions of its books, in portable document format (pdf), will be available at its website, for free:

“As of June 2, 2011, all PDF versions of books published by the National Academies Press (NAP) will be downloadable to anyone free of charge.

That’s more than 4,000 books plus future reports produced by NAP – publisher for the National Academy of Sciences, National Academy of Engineering, Institute of Medicine, and National Research Council.”

Important works on forensic evidence, asbestos, dioxin, beryllium, research ethics, and data sharing published by the NAP, for the IOM or NRC, are now available for free. The NAP charged upwards of $40 or 50 for some of these books previously.

This summer, the NRC’s Committee on Science, Technology and Law will release the Third Edition of the Reference Manual on Scientific Evidence, previously prepared by the Federal Judicial Center. See http://sites.nationalacademies.org/PGA/stl/development_manual/index.htm

Posted in Rule 702 | No Comments »

Statistical Power in the Academy

June 1st, 2011

Previously I have written about the concept of statistical power and how it is used and abused in the courts. See here and here.

Statistical power was discussed in both the chapters on statistics and on epidemiology in the Second Edition of The Reference Manual on Scientific Evidence. In my earlier posts, I pointed out that the chapter on epidemiology provided some misleading, outdated guidance on the use of power. See Michael D. Green, D. Michal Freedman, and Leon Gordis, “Reference Guide on Epidemiology,” in Federal Judicial Center, The Reference Manual on Scientific Evidence 333, 362-63 (2ed. 2000) (recommending use of power curves to assess whether failure to achieve statistical significance is exonerative of the exposure in question). This chapter suggests that “[t]he concept of power can be helpful in evaluating whether a study’s outcome is exonerative or inconclusive.” Id.; see also David H. Kaye and David A. Freedman, Reference Guide on Statistics,” Federal Judicial Center, Reference Manual on Scientific Evidence 83, 125-26 (2ed. 2000).

The fact of the matter is that power curves are rarely or never used in contemporary epidemiology, and post-hoc power calculations have been discouraged and severely criticized for a long time. After the data are collected, the appropriate method to evaluate the “resolving power” of a study is to examine the confidence interval around the study’s estimate of risk size. These confidence intervals allow a concerned reader to evaluate what can reasonably ruled out (on the basis of random variation only) by the data in a given study. Post-hoc power calculations or considerations fail to provide meaningful consideration because they require a specified alternative hypothesis.

Twenty-five years ago, the use of post-hoc power was thoughtfully put in the dust bin of statistical techniques in the leading clinical medical journal:

“Although power is a useful concept for initially planning the size of a medical study, it is less relevant for interpreting studies at the end. This is because power takes no account of the actual results obtained.”

***

“[I]n general, confidence intervals are more appropriate than power figures for interpreting results.”

Richard Simon, “Confidence intervals for reporting results of clinical trials,” 105 Ann. Intern. Med. 429, 433 (1986) (internal citation omitted).

An accompanying editorial by Ken Rothman reinforced the guidance given by Simon:

“[Simon] rightly dismisses calculations of power as a weak substitute for confidence intervals, because power calculations address only the qualitative issue of statistical significance and do not take account of the results already in hand.”

Kenneth J. Rothman, “Significance Questing,” 105 Ann. Intern. Med. 445, 446 (1986)

These two papers must be added to the 20 consensus statements, textbooks, and articles I previously cited. See Schachtman, Power in the Courts, Part Two (2011).

The danger of the Reference Manual’s misleading advice is illustrated in a recent law review article by Professor Gold, of the Rutgers Law School, who asks “[w]hat if, as is frequently the case, such study is possible but of limited statistical power?” Steve C. Gold, “The ‘Reshapement’ of the False Negative Asymmetry in Toxic Tort Causation, 37 William Mitchell L. Rev. 101, 117 (2011) (available at http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1797826).

Never mind for the moment that Professor Gold offers no empirical evidence to support his assertion that studies of limited statistical power are “frequently” used in litigation. Gold critically points to Dunn v. Sandoz Pharmaceuticals Corp., 275 F. Supp. 2d 672, 677–81, 684 (M.D.N.C. 2003), a Parlodel case in which case plaintiff relied upon a single case-control study that found an elevated odds ratio (8.4), which was not statistically significant. Gold at 117. Gold complains that “a study’s limited statistical power, rather than the absence of a genuine association, may lead to statistically insignificant results that courts treat as disproof of causation, particularly in situations without the large study samples that result from mass exposures.” Id. Gold goes on to applaud two cases for emphasizing consideration of post-hoc power. Id. at 117 & n. 80 – 81 (citing Smith v. Wyeth-Ayerst Labs. Co., 278 F. Supp. 2d 684, 692 – 93 (W.D.N.C. 2003) (“[T]he concept of power is key because it’s helpful in evaluating whether the study‘s outcome . . . is exonerative or inconclusive.”), and Cooley v. Lincoln Elec. Co., 693 F. Supp. 2d 767, 774 (N.D. Ohio 2010) (prohibiting expert witness from opining that epidemiologic studies are evidence of no association unless the witness “has performed a methodologically reliable analysis of the studies’ statistical power to support that conclusion”).

What of Professor Gold’s suggestion that power should be considered in evaluating studies that do not have statistically significant outcomes of interest? See id. at 117. Not only is Gold’s endorsement at odds with sound scientific and statistical advice, but his approach reveals a potential hypocrisy when considered in the light of his criticisms of significance testing. Post-hoc power tests ignore the results obtained, including the variance of the actual study results, and they are calculated based upon a predetermined arbitrary measure of Type I error (alpha) that is the focus of so much of Gold’s discomfort with statistical evidence. Of course, power calculations also are made on the basis of arbitrarily selected alternative hypotheses, but this level of arbitrariness seems not to disturb Gold so much.

Where does the Third Edition of the Reference Manual on Scientific Evidence come out on this issue? The Third Edition is not yet published, but Professor David Kaye has posted his chapter on statistics on the internet. David H. Kaye & David A. Freedman, “Reference Guide on Statistics,” chapter 5. http://www.personal.psu.edu/dhk3/pubs/11-FJC-Ch5-Stat.pdf (David Freedman died in 2008, after the chapter was submitted to the National Academy of Sciences for review; only Professor Kaye responded to the Academy’s reviews).

The chapter essentially continues the Second Edition’s advice:

“When a study with low power fails to show a significant effect, the results may therefore be more fairly described as inconclusive than negative. The proof is weak because power is low. On the other hand, when studies have a good chance of detecting a meaningful association, failure to obtain significance can be persuasive evidence that there is nothing much to be found.”

Chapter 5, at 44-46 (citations and footnotes omitted).

The chapter’s advice is not, of course, limited to epidemiologic studies, where a risk ratio or a risk difference is typically reported with an appropriate confidence interval. In the broad generality of considering all statistical tests, some of which do not report a measure of “effect size,” and the variability of the sample statistic, the chapter’s advice is fine. But, as we can see from Professor Gold’s discussion and case review, the advice runs into trouble when measured against the methodological standards for evaluating an epidemiologic study’s results when confidence intervals are available. Gold’s assessment of the cases is considerably skewed by his failure to recognize the inappropriateness of post-hoc power assessments of epidemiologic studies.

Posted in Rule 702 | No Comments »

Manufactured Certainty

May 27th, 2011

With the help of Selikoff’s Lobby, the anti-asbestos zealots have created a false, manufactured certainty about various asbestos issues. The manufacturing of faux certainty has taken place with respect to the history of knowledge about asbestos, as well as to the current state of knowledge about asbestos hazards.

The Selikoff lobby exercised a great deal of influence on regulators and scientists. The Lobby was able to bully many scientists and policy makers into adopting a position that held all asbestos mineral fiber types as relatively equal in their potency to cause disease. The Lobby accomplished this by suppressing evidence of past use of amphibole asbestos, and by overstating the hazards of chrysotile asbestos.

In the past, I have marshaled evidence of Selikoff’s activities as a crocidolite denier. But was there really a controversy among honest scientists outside the Lobby?

Of course, there was and there is, but the Lobby has done a good job of branding the contrarians as tools of industry. It is important, therefore, to come to terms with evidence that scientists without connections to industry took similar positions.

For many years, starting in the late 1970s, Dr. Gerrit Schepers was a mainstay of the plaintiffs’ state-of-the-art case against asbestos mining and manufacturing companies in asbestos personal litigation. Dr. Schepers testified as a hired expert witness for plaintiffs near and far. I encountered and crossexamined Dr. Schepers on several occasions, for different clients. He was a fascinating witness, filled with contradictions and mixed motives. In one particularly horrible mesothelioma case (Hill v. Carey Canada), I confronted Dr. Schepers with his own publication, from 1973, in which he largely exonerated chrysotile as a carcinogen. Dr. Schepers twisted and turned, but he really had no where to go to avoid the full force of his own statements. This publication is worth revisiting as an historical document, to show that there was a good deal of dissent from the Lobby’s positions, at least until the asbestos personal injury and property damage litigations mushroomed out of control in the early 1980s.

Here is what Dr. Schepers wrote, in 1973, while an employee of the United States government (Chief of the Medical Service, Veterans Administration, Lebanon, Pa.):

“There are marked differences between the capacities of the individual classes of silicate minerals to provoke responses in human and animal tissues. There also are major misconceptions as to what these substances can do when inhaled by man or other mammals. Two of the most extreme of these are (1) that all siliceous minerals are equally pathogenic and (2) that there is even the least semblance between the effects of the asbestiform and the non-asbestiform silicates.”

Gerrit W. H. Schepers, M.D. D.Sc., “The Biological Action of Talc and Other Silicate Minerals,” at 54, in Aurel Goodwin, Proceedings of the symposium on talc: U.S. Bureau of Mines; Information Circular 8639 (1974) [available at http://www.scribd.com/doc/56461314]. The symposium was sponsored by the United States Department of the Interior, in May 1973. Recall that the dispute of non-asbestiform amphibole health effects was very much at issue in the Reserve Mining case, and the trial proceedings were about to start when Dr. Schepers delivered his paper, in 1973. Members of the Lobby, from Selikoff on down, were very much involved in the Reserve Mining case. See U.S. Environmental Protection Agency v. Reserve Mining Co., 514 F.2d 492 (8th Cir. 1975) (en banc).

“Is chrysotile a carcinogen? This is a very perplexing question. A crescendo of popular opinion has sought to incriminate chrysotile. This author remains unconvinced. The main premise for carcinogenicity stems from epidemiological observation of employees of the insulation and shipbuilding industries. In both these industries there has been in the past considerable exposure of pipe laggers to asbestos dust. Only in recent decades, however, have these insulation bats been composed predoninantly of chrysotile. In former years crocidolite and amosite were important components.

***

Finally, it should be pointed out that the role of cigarette smoking has not been satisfactorily discounted in the referenced epidemiological studies of lung cancer among insulation workers. In some groups reported an excess prevalence of lung cancer was not demonstrable when cigarette smoking was taken into consideration. Epidemiological surveys of chrysotile workers in Quebec showed no excess of lung cancer. A review of pleural mesothiliomatosis in Canada also failed to focus attention on Quebec or any other center where chrysotile industries are concentrated.”

Id. at 70.

That was in 1973, but within a few years, Dr. Schepers was coopted by the asbestos plaintiff industry, which manufactured lawsuits and epistemic certainty about the hazards of all asbestos minerals. The rest is “history.”

Interestingly, another would-be historian in the asbestos litigation, Dr. David S. Egilman, has written a paper, highly critical of W.R. Grace, based in part on another presentation given at the 1973 symposium, referenced above. David Egilman, Wes Wallace, and Candace Hom, Corporate corruption of medical literature: Asbestos studies concealed by W. R. Grace & Co., 6 Accountability in Research 127 (1998)(citing a paper in the same volume by Dr. William E. Smith, “Experimental studies on biological effects of tremolite talc on hamsters.”). Egilman’s paper was available at is website, http://www.egilman.com/Documents/publications/Wr_Grace.pdf The paper by Dr. Schepers no doubt missed Egilman’s attention, even though it follows immediately after Dr. Smith’s contribution.

Posted in Conflicts of Interest, Historian Testimony | No Comments »

Sub-group Analyses in Epidemiologic Studies — Dangers of Statistical Significance as a Bright-Line Test

May 17th, 2011

Both aggregation and disaggregation of outcomes poses difficult problems for statistical analysis, and for epidemiology. If outcomes are bundled into a single composite outcome, there has to be some basis for the bundling to make sense. Even so, a composite outcome, such as all cardiovascular disease events, could easily hide an association in a component outcome. For instance, studies of a drug under scrutiny may show no increased risk for all cardiovascular events, but closer inspection may show an increased risk for heart attacks while also showing a decreased risk for strokes.

The opposite problem arises when studies report multiple subgroups. The opportunity for post hoc data mining runs rampant, and the existence of multiple subgroups means that the usual level of statistical significance becomes ineffective for ruling out chance as an explanation for an increased or decreased risk in a subgroup. This problem is well known and extensively explored in the epidemiology literature, but it receives no attention in the Federal Judicial Center’s current Reference Manual on Scientific Evidence. I hope that the authors of the Third Edition, which is due out in a few months, give some attention to the problem of subgroup analysis in epidemiology. This seems to be an area where judges need a good deal of assistance, and where the Reference Manual lets them down.

Litigation tends to be a fertile field for the data dredging or the Texas Sharp shooters’ approach to epidemiology. (The Texas Sharp shooter shoots first and draws the target later.) When studies look at many outcomes, or many subgroups, chance alone will lead to results that have p-values less than the usual level for statistical significance (p < 0.05). Accepting a result as “significant” when there is a multiplicity of testing or comparisons resulting from subgroup analyses is a form of “data torturing.” Mills, “Data Torturing,” 329 New Engl. J. Med. 1196, 1196 (1993)(“If you torture the data long enough, they will confess.”).

The multiple testing or comparison issue arises in both cohort and case-control studies. Cohort studies have the ability to look at cancer morbidity or mortality at 20 different organs, with multiple histological subtypes for each cancer. There are hundreds of diseases, by World Health Organization disease codes, which can be a possible outcome in a cohort study. The odds are very good that several disease outcomes will be significantly elevated or decreased by chance alone. Similarly, in a case-control study, participants with the outcome of interest can be questioned about hundreds of lifestyle and exposure variables. Again, the finding of a “risk factor,” with statistical significance is not very compelling under these circumstances.

The problem of subgroup analyses is exacerbated by defense counsel’s emphasis on statistical significance as a “bright-line” test. When subgroup analyses yield a statistically significant result, at the usual p < 0.05, which they will often do by chance alone, plaintiffs’ counsel obtain a “gotcha” moment. Having built up the importance of statistical significance, defense counsel are hard pressed to dismiss the “significant” finding, even though study design makes it highly questionable if not downright meaningless.

Although the Reference Manual ignores this recurrent problem, several authors have issued severe alerts to the issue. For instance, Lisa Bero, who writes frequently on science and the law issues, admonishes:

“Specifying subgroup analysis after data collection for the review has already begun can be a ‘fishing expedition’ or “data dredging” for statistically significant results and is not appropriate.”

L. Bero, “Evaluating Systematic Reviews and Meta-Analyses,” J. L. & Policy 569, 576 (2006).

Eggers and Davey Smith, two well-respected English authors, who write about methodological issues in epidemiology, warn:

“Similarly, unplanned data-driven subgroup analyses are likely to produce spurious results.”

Matthias Egger & George Davey Smith, “Principles of and procedures for systematic reviews,” 24 chap. 2, in M. Egger, G. Davey Smith, D. Altman, eds., Systematic Reviews in Health Care: Meta-Analysis in Context (2d ed. 2001).

Stewart and Parmar explain the genesis of the problem and the result of diluting the protection that statistical significance usually provides against Type I errors:

“In general, the results of these subgroup analyses can be very misleading owing to the very high probability that any observed differences is due solely to chance.⁸ For example, if 10 subgroup analyses are carried out, there is a 40% chance of finding at least one significant false-positive effect (5% significance level). Further, when the results of subgroup analyses are reported, often only those that have yielded a significant result are presented, without noting that many other analyses have been performed.”

Stewart and Parmar, “Bias in the Analysis and Reporting of Randomized Controlled Trials,” 12 Internat’l J. Tech. Assessment in Health Care 264, 271 (1996)

“Such data dredging must be avoided and subgroup analyses should be limited to those that are specified a priori in the trial protocol.”

Id. at 272.

“Readers and reviewers should be aware that subgroup analyses, exploratory or otherwise, are likely to be particularly unreliable in situations where no overall effect of treatment has been observed. In this case, if one subgroup exhibits a particularly positive effect of treatment, then another subgroup has to have a counteracting negative effect.”

* * *

“Consequently, perhaps the most sensible advice to readers and reviewers is to be very skeptical about the results of subgroup analyses.”

Id. See also Sleight, “Subgroup analyses in clinical trials – – fun to look at, but don’t believe them,” 1 Curr. Control Trials Cardiovasc. Med. 25 (2000) (“Analysis of subgroup results in a clinical trial is surprisingly unreliable, even in a large trial. This is the result of a combination of reduced statistical power, increased variance and the play of chance. Reliance on such analyses is likely to be erroneous, and hence harmful, than application of the overall proportional (or relative) result in the whole trial to the estimate of absolute risk in that subgroup. Plausible explanations can usually be found for effects that are, in reality, simply due to the play of chance. When clinicians believe such subgroup analyses, there is a real damage of harm to the individual patient.”)

These warnings and admonitions are important caveats to statistical significance. In emphasizing the importance of statistical significance in evaluating statistical evidence, defense lawyers are sometimes unwittingly hoisted with their own petard, in the form of studies that have results that meet the usual p-value threshold of lower than 5%. Courts see these defense lawyers as engaged in special pleading when counsel argues that study multiplicity requires changing the p-value threshold to preserve the desired rate of Type I error, but that is exactly what must be done.

A few years ago, the New England Journal of Medicine published an article that detailed the problem and promulgated guidelines for avoiding the worst abuses. R. Wang, S. Lagakos, J. H. Ware, et al., “Statistics in Medicine — Reporting of Subgroup Analyses in Clinical Trials,” 357 New Engl. J. Med. 2189 (2007). Wang and colleagues provide some important insights for how subgroup analyses can lead to increased rates of Type I errors, and they provide guidelines for authors on appropriate descriptions of subgroup analyses:

“However, subgroup analyses also introduce analytic challenges and can lead to overstated and misleading results.”

Id. at 2189a.

“When multiple subgroup analyses are performed, the probability of a false positive finding can be substantial.”

Id. at 2190a.

“There are several methods for addressing multiplicity that are based on the use of more stringent criteria for statistical significance than the customary P < 0.05.”

Id. at 2190b.

“A pre-specified subgroup analysis is one that is planned and documented before any examination of the data, preferably in the study protocol.”

Id. at 2190b.

“Post hoc analyses refer to those in which the hypotheses being tested are not specified before any examination of the data. Such analyses are of particular concern because it is often unclear how many were undertaken and whether some were motivated by inspection of the data. However, both pre-specified and post hoc subgroup analyses are subject to inflated false positive rates arising from multiple testing. Investigators should avoid the tendency to pre-specify many subgroup analyses in the mistaken belief that these analyses are free of the multiplicity problem.”

Id. at 2190b.

“When properly planned, reported, and interpreted, subgroup analyses can provide valuable information.”

Id. at 2193b.

Although Wang and colleagues take their primary aim at the abuse of subgroup analyses in randomized clinical trials, they make clear that the abuse is equally present in observational studies:

“In other settings, including observational studies, we encourage complete and thorough reporting of the subgroup analyses in the spirit of the guidelines listed.”

Id. at 2193b.

Wang and colleagues provide some very specific guidelines for reporting subgroup analyses. These guidelines are a helpful source for helping courts make sober assessments of results from subgroup analyses.

Recently, another guideline initiative, STROBE, in the field of observational epidemiology provided similar guidance to authors and journals for reporting subgroup analyses:

“[M]any debate the use and value of analyses restricted to subgroups of the study population. Subgroup analyses are nevertheless often done. Readers need to know which subgroup analyses were planned in advance, and which arose while analyzing the data. Also, it is important to explain what methods were used to examine whether effects or associations differed across groups … .”

Jan P. Vandenbroucke, Erik von Elm, Douglas G. Altman, Peter C. Gøtzsche, Cynthia D. Mulrow, Stuart J. Pocock, Charles Poole, James J. Schlesselman, and Matthias Egger, for the STROBE Initiative, “Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): Explanation and Elaboration,” 18 Epidemiology 805, 817 (2007).

“There is debate about the dangers associated with subgroup analyses, and multiplicity of analyses in general. In our opinion, there is too great a tendency to look for evidence of subgroup-specific associations, or effect-measure modification, when overall results appear to suggest little or no effect. On the other hand, there is value in exploring whether an overall association appears consistent across several,

preferably pre-specified subgroups especially when a study is large enough to have sufficient data in each subgroup. A second area of debate is about interesting subgroups that arose during the data analysis. They might be important findings, but might also arise by chance. Some argue that it is neither possible nor necessary to inform the reader about all subgroup analyses done as future analyses of other data will tell to what extent the early exciting findings stand the test of time. We advise authors to report which analyses were planned, and which were not … . This will allow readers to judge the implications of multiplicity, taking into account the study’s position on the continuum from discovery to verification or refutation.”

Id. at 826-27.

Bibliography

E. Akl, M. Briel, J.J. You, et al., “LOST to follow-up Information in Trials (LOST-IT): a protocol on the potential impact,” 10 Trials 40 (2009).

Susan Assmann, Stuart Pocock, Laura Enos, Linda Kasten, “Subgroup analysis and other (mis)uses of baseline data in clinical trials,” Lancet 2000; 355: 1064–69.

M. Bhandari, P.J. Devereaux, P. Li, et al., “Misuse of baseline comparison tests and subgroup analyses in surgical trials,” 447 Clin. Orthoped. Relat. Res. 247 (2006).

S. T. Brookes, E. Whitely, M. Egger, et al., “Subgroup analyses in randomized trials: risks of subgroup-specific analyses; power and sample size for the interaction test,” 57 J. Clin. Epid. 229 (2004).

A-W Chan, A. Hrobjartsson, K.J. Jorgensen, et al., “Discrepancies in sample size calculations and data analyses reported in randomised trials: comparison of publications with protocols,” 337 Brit. Med. J. a2299 (2008).

L. Cui, H.M. Hung, S.J. Wang, et al., “Issues related to subgroup analysis in clinical trials,” 12 J. Biopharm. Stat. 347 (2002).

Matthias Egger & George Davey Smith, “Principles of and procedures for systematic reviews,” chap. 2, in M. Egger, G. Davey Smith, D. Altman, eds., Systematic Reviews in Health Care: Meta-Analysis in Context (2d ed. 2001).

J. Fletcher, “Subgroup analyses: how to avoid being misled,” 335 Brit. Med. J. 96 (2007).

Nick Freemantle,”Interpreting the results of secondary end points and subgroup analyses in clinical trials: should we lock the crazy aunt in the attic?” 322 Brit. Med. J. 989 (2001).

G. Guyatt, P.C. Wyer, J. Ioannidis, “When to Believe a Subgroup Analysis,” in G. Guyatt, et al., eds., User’s Guide to the Medical Literature: A Manual for Evidence-Based Clinical Practice 571-83 (2008).

J. Hasford, P. Bramlage, G. Koch, W. Lehmacher, K. Einhäupl, and P.M. Rothwell, “Inconsistent trial assessments by the National Institute for Health and Clinical Excellence and IQWiG: standards for the performance and interpretation of subgroup analyses are needed,” 63 J. Clin. Epidem. 1298 (2010).

J. Hasford, P. Bramlage, G. Koch, W. Lehmacher, K. Einhäupl, and P.M. Rothwell, “Standards for subgroup analyses are needed? We couldn’t agree more,” 64 J. Clin. Epidem. 451 (2011).

R. Hatala, S. Keitz, P. Wyer, et al., “Tips for learners of evidence-based medicine: 4. Assessing heterogeneity of primary studies in systematic reviews and whether to combine their results,” 172 Can. Med. Ass’n J. 661 (2005).

A.V. Hernandez, E.W. Steyerberg, G.S. Taylor, et al., “Subgroup analysis and covariate adjustment in randomized clinical trials of traumatic brain injury: a systematic review,” 57 Neurosurgery 1244 (2005).

A.V. Hernandez, E. Boersma, G.D. Murray, et al., “Subgroup analyses in therapeutic cardiovascular clinical trials: are most of them misleading?” 151 Am. Heart J. 257 (2006).

K. Hirji & M. Fagerland, “Outcome based subgroup analysis: a neglected concern,” 10 Trials 33 (2009).

Stephen W. Lagakos, “The Challenge of Subgroup Analyses — Reporting without Distorting,” 354 New Engl. J. Med. 1667 (2006).

C.M. Martin, G. Guyatt, V. M. Montori, “The sirens are singing: the perils of trusting trials stopped early and subgroup analyses,” 33 Crit. Care Med. 1870 (2005).

D. Moher, K. Schulz, D. Altman, et al.,“The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials,” 357 Lancet 1191 (2001).

V.M. Montori, R. Jaeschke, H.J. Schunemann, et al., “Users’ guide to detecting misleading claims in clinical research reports,” 329 Brit. Med. J. 1093 (2004).

A.D. Oxman & G.H. Guyatt, “A consumer’s guide to subgroup analyses,” 116 Ann. Intern. Med. 78 (1992).

A. Oxman, G. Guyatt, L. Green, et al., “When to believe a subgroup analysis,” in G. Guyatt, et al., eds., User’s Guide to the Medical Literature: A Manual for Evidence-Based Clinical Practice 553-65 (2008).

S. Pocock, M. D. Hughes, R.J. Lee, “Statistical problems in the reporting of clinical trials: A survey of three medical journals,” 317 New Engl. J. Med. 426 (1987).

S. Pocock, S. Assmann, L. Enos, et al., “Subgroup analysis, covariate adjustment and baseline comparisons in clinical trial reporting: current practice and problems,” 21 Statistics in Medicine 2917 (2002).

Peter Rothwell, “Subgroup analysis in randomised controlled trials: importance, indications, and interpretation,” 365 Lancet 176 (2005).

Kenneth Schulz & David Grimes, “Multiplicity in randomised trials II: subgroup and interim analyses,” 365 Lancet 1657 (2005).

Sleight, “Subgroup analyses in clinical trials – – fun to look at, but don’t believe them,” 1 Curr. Control Trials Cardiovasc. Med. 25 (2000).

Reuel Stallones, “The Use and Abuse of Subgroup Analysis in Epidemiological Research,” 16 Prev. Med. 183 (1987).

Stewart & Parmar, “Bias in the Analysis and Reporting of Randomized Controlled Trials,” 12 Internat’l J. Tech. Assessment in Health Care 264, 271 (1996).

Xin Sun, Matthias Briel, Jason Busse, Elie A. Akl, John J .You, Filip Mejza, Malgorzata Bala, Natalia Diaz-Granados, Dirk Bassler, Dominik Mertz, Sadeesh K Srinathan, Per Olav Vandvik, German Malaga, Mohamed Alshurafa, Philipp Dahm, Pablo Alonso-Coello, Diane M Heels-Ansdell, Neera Bhatnagar, Bradley C. Johnston, Li Wang, Stephen D. Walter, Douglas G. Altman, and Gordon Guyatt, “Subgroup Analysis of Trials Is Rarely Easy (SATIRE): a study protocol for a systematic review to characterize the analysis, reporting, and claim of subgroup effects in randomized trials,” 10 Trials 1010 (2009).

A. Trevor & G. Sheldon, “Criteria for the Implementation of Research Evidence in Policy and Practice,” in A. Haines, ed., Getting Research Findings Into Practice 11 (2d ed. 2008).

Erik von Elm & Matthias Egger, “The scandal of poor epidemiological research Reporting guidelines are needed for observational epidemiology,” 329 Brit. Med. J. 868 (2004).

R. Wang, S. Lagakos, J. H. Ware, et al., “Statistics in Medicine — Reporting of Subgroup Analyses in Clinical Trials,” 357 New Engl. J. Med. 2189 (2007).

S. Yusuf, J. Wittes, J. Probstfield, et al., “Analysis and interpretation of treatment effects in subgroups of patients in randomized clinical trials,” 266 J. Am. Med. Ass’n 93 (1991).

Posted in Rule 702 | 74 Comments »

The Law’s Obsession with Warnings

May 11th, 2011

Professor Beth J. Rosenberg is Assistant Professor, in the Department of Public Health & Community Medicine, in Tufts University School of Medicine, Boston, Massachusetts. Rosenberg is an unabashed activist. She is driven by concerns that humans are ruining the environment and poisoning themselves. She is a champion of workers’ safety and workers’ rights. So when she writes about her personal experience with the lack of interest among workers in the hazards of silica, we all can learn something about whether the law’s obsession with warnings makes sense.

In 2003, Rosenberg wrote an article about her experiences in trying to have silica added to the list of substances regulated under the Massachusetts’ Toxics Use Reduction Act (TURA). Beth Rosenberg, “Second Thoughts About Silicosis,” 13 New Solutions 223 (2003) (http://www.ncbi.nlm.nih.gov/pubmed/17208725). Working with support from the Environmental League of Massachusettes and the Massachusettes Public Health Association, Rosenberg petitioned to have silica added to the TURA list of substances, in part out of a desire to help fuel a ban on abrasive blasting with silica in Massachusettes. She figured that by piggybacking on the environmental movement, or riding “the green wave,” as she put it, the state’s environmental laws could be used to help control occupational exposures.

Rosenberg’s ideals and aspirations ran into the wall of worker expectations and needs. They did not want abrasive blasting banned; they wanted stronger enforcement from OSHA, and better respirators. Rosenberg admits that the workers were pursuing a path that was not her goal, and she learned that, at legislative hearings, she needed “to take tighter control of the scripts of any hearings that I’m orchestrating.” Id. at 227.

Rosenberg worked with the Painters’ union to study substitutes for silica in abrasive blasting. Motivated by a recognition that “[s]ilica-related disease is completely preventable,” id. at 224, she hoped to move them towards supporting a ban on silica for abrasive blasting. After several years of this work, however, Rosenberg decided to give up on her silica mission. Her experience is instructive for correcting the misapplication of “failure to warn” products liability law to the use of a raw material such as silica in the workplace:

“The main point here is that the men I’ve interviewed are not terribly concerned about silica dust. They care about being treated decently and respectfully by their bosses. They’re concerned about being encouraged to work too fast to work safely. They care about lead dust, particularly bringing it home to their families, so they get really angry when the foreman wants to lock up the yard at five o’clock and doesn’t leave them enough time to shower and change their clothes. They feel that they are expendable. And although most are fully aware of silica’s dangers, silica is not a top priority for them. The silica agenda was set by some physicians and health professionals who are outraged that anybody is still dying of this 100 percent preventable disease. This is understandable, and I am one of those people, but I’m not sure this is the best way to be of service. I see that there are other, more pressing issues than silica.

I’ve chosen to serve working people, and yet they’ve had little or no role in setting the research agenda. Not only is this unrewarding for me, but it’s also a bad political strategy because you need a lot of support and collaboration to accomplish anything—even when everyone agrees that action is required—and interest in silica is tepid among the people most affected. This may not be true in other trades or in other countries, but it is true with abrasive blasters. And I stress that is not an awareness problem; they know breathing dust is bad for them, but it’s just not their top concern, and I can see why. So, henceforth, I’m going to let the community I choose to serve set the research agenda, and I will offer my assistance in their battles. That to me is the best way to do public health.”

Id. at 229 (emphasis added). Rosenberg’s epiphany should lead to some thoughtful re-evaluation of how the law of products liability is applied to the use of a natural material such as crystalline silica. While Professor Rosenberg was working with the Painters’ Union, and having her “Second Thoughts about Silicosis,” plaintiffs’ lawyers were screening, scheming, and suing for silicosis among the same union’s members. If only plaintiffs’ law firms took heed of Professor Rosenberg’s lessons, and stopped signing up sand blasters under the paternalistic pretense that the law must provide a remedy for the alleged failure to warn. The faux historians of silicosis, with their conspiratorial theories, could learn a great deal from Professor Rosenberg, as well.

Posted in Historian Testimony | No Comments »

The opinions, statements, and asseverations expressed on Tortini are my own, or those of invited guests, and these writings do not necessarily represent the views of clients, friends, or family, even when supported by good and sufficient reason.