Pharmaceutical manufacturers are particularly vulnerable to securities fraud claims arising from the manufacturers’ pronouncements about safety or efficacy, the evidence for which is often statistical in nature.  Safety claims may involve complex data sets, both from observational studies and clinical trials.  Efficacy claims are typically based upon clinical trial data.

Publicly traded manufacturers may find themselves caught between competing securities regulations.  In evaluating safety or efficacy data, manufacturers will often consult with an outside science advisory board, or report to regulatory agencies.  Securities regulations specify that any disclosure of confidential inside information to an outsider triggers an obligation of prompt public disclosure of that information.[1]  Companies also routinely seek to keep investors informed of research and marketing developments.  Generally, manufacturers will make their public disclosures through widely circulated press releases.[2]  Not surprisingly, disgruntled investors may challenge the accuracy of the press releases, when the product or drug turns out to be less efficacious or more harmful than represented in the press release.  These challenges, brought under the securities laws, often are maintained in parallel to product liability actions, sometimes in the same multi-district litigation.

Securities laws require accurate disclosure of all material information.[3]  Rule 10b-5 of the Securities Exchange Commission (SEC) prohibits any person from making “any untrue statement of material fact” or from omitting “a material fact necessary in order to make the statements made, in light of the circumstances under which they were made, not misleading.”[4]

A prima facie case of securities fraud requires that plaintiff allege and establish, among other things, a material misrepresentation or omission.[5]  The obligations to speak and to speak accurately have opened manufacturers to second guessing in their analyses of safety and efficacy data.  In most securities fraud cases, courts have given manufacturers a wide berth by rejecting differences in opinions about the proper interpretation of studies as demonstrating fraud under the securities regulations.[6]  This latitude has been given both in judgment of what test procedures to use, as well as in how best to interpret data.[7]   In Padnes v. Scios Nova Inc., the manufacturer was testing a drug for treatment of acute kidney failure.  Scios Nova issued a press release after its phase II trial, to announce a statistically significant reduction in patients’ need for dialysis.  When the early phase III results failed to confirm this result, plaintiffs sued Scios Nova for not disclosing the lack of statistically significant outcomes on other measures of kidney function, as well as for its interpretation of dialysis results as statistically significant.[8]  The trial court dismissed the complaint.[9]

Several securities fraud cases have turned on investor dissatisfaction on how companies interpreted clinical trial subgroup data.  In Noble Asset Management v. Allos Therapeutics, Inc.,[10] the company issued a press release, noting no statistically significant increase overall in survival advantage from a drug for breast cancer, but also noting a statistically significant increased survival in a non-prespecified subgroup of patients with metastatic breast cancer.[11] The plaintiff investors claimed that the company should have disclosed that the FDA was unlikely to approve an indication based upon an ad hoc subgroup analysis, but the trial court rejected this claim because FDA policy on drug approvals is public and well known.[12] The plaintiffs also complained that the press release referred to statistically significant results from a Cox multiple regression analysis rather than the log-rank (non-parametric survival) analysis required by FDA. The trial court rejected this claim as well, opining that the analysis was not misleading when the company correctly reported the raw data and the results of the Cox multiple regression analysis.[13]

Two recent appellate decisions emphasize the courts’ unwillingness to scrutinize the contested statistical methodology that underlies plaintiffs’ claims of misrepresentation.  In In re Rigel Pharmaceuticals, Inc. Securities Litigation, the plaintiff investors were dissatisfied, not with reporting of subgroups, but rather with the failure of the company to report geographic subgroup results, as well as its use of allegedly improper statistical tests and its failure to account for multiple comparisons.[14]

The Ninth Circuit affirmed the dismissal of a complaint.  The appellate court held that allegations of “statistically false p-values” were not sufficient; plaintiffs must allege facts that explain why the difference between two statements “is not merely the difference between two permissible judgments, but rather the results of a falsehood.”[15] Alleging that a company should have used a different statistical method to analyze the data from its clinical trial is not sufficient to raise an issue of factual falsity under the securities fraud statute and regulations.[16]  The Court explained that the burden was on plaintiffs to plead and prove that the difference between two statistical statements “is not merely the difference between two permissible judgments, but rather the result of a falsehood.”[17] The Court characterized the plaintiffs’ allegations to be about judgments of which statistical tests or methods are appropriate, and not about false statements.  Furthermore, the Court emphasized that the company’s statistical method was called for in the trial protocol, and was selected before the data were unblinded and provided to the company.[18]

In Kleinman v. Elan Corporation[19], the Second Circuit affirmed the dismissal of a securities fraud class action against two pharmaceutical joint venturers, which issued a challenged press release on interim phase II clinical trial results for bapineuzumab, a drug for mild- to moderate-Alzheimer’s disease.  The press release at issue announced “top line” findings and promised a full review at an upcoming international conference.[20]  According to the release, the clinical trial data did not show a statistically significant benefit on the primary efficacy end point, but “[p]ost-hoc analyses did show statistically significant and clinically meaningful benefits in important subgroups.”[21]

The plaintiffs in Kleinman complained that the clinical trial had started with crucial imbalances between drug and placebo arms, thus indicating a failure in randomization, and that the positive results had come from impermissible post-hoc subgroup analyses.[22]  The appellate court appeared not to take the randomization issue seriously, and rejected the notion that statements can be false when they represent a defendant company’s reasonable interpretation of the data, even when the interpretation later turns out to be shown to be false[23]:

“At bottom, Kleinman simply has a problem with using post-hoc analyses as a methodology in pharmaceutical studies.  Kleinman cites commentators who liken post-hoc analyses to moving the goal posts or shooting an arrow into the wall and then drawing a target around it. Nonetheless, when it is clear that a post-hoc analysis is being used, it is understood that those results are less significant and should have less impact on investors.  Our job is not to evaluate the use of post-hoc analysis in the scientific community; the FDA has already done so.”

In United States v. Harkonen[24], the government turned the law of statistical analyses in securities fraud on its head, when it prosecuted a pharmaceutical company executive for his role in issuing a press release on clinical trial data. The jury acquitted Dr. Harkonen on a charge of misbranding[25], but convicted on a single count of wire fraud.[26] Dr. Harkonen’s crime?  Bad statistical practice.

The government conceded that the data represented in the press release were accurate, as were the calculated p-values.  The chargeable offense lay in Dr. Harkonen’s describing the clinical trial results as “demonstrating a survival benefit” of the biological product (interferon γ-1b) in a clinical trial subgroup of patients with mild- to moderate-idiopathic pulmonary fibrosis.  The p-value for the subgroup was 0.004, with an effect size of 70% reduction in mortality.  The subgroup, however, was not prespecified, and was not clearly labeled as a post-hoc analysis.  The trial had not achieved statistical significant on its primary end point.

In prosecuting Dr. Harkonen, the government offered no expert witness opinion.  Instead, it relied upon a member of the clinical trial’s data safety monitoring board, who advanced a strict, orthodox view that if the primary end point of a trial “failed,” then the data could not be relied upon to infer any meaningful causal connection within secondary end points, let alone non-prespecified end points.  The prespecified survival secondary end point showed a 40 percent reduction in mortality, p = 0.08 (which shrank to 0.055 on an intent-to-treat analysis). The press release also relied upon a previous small clinical trial that showed a benefit in survival at five years, with the therapy group at 77.8%, compared with 16.7% in the control groups, p = 0.009.

The trial court accepted the government’s claim that p-values less than 0.05 were something of “magic numbers,”[27] and rejected post-trial motions for accquittal. Dr. Harkonen’s use of “demonstrate” to describe a therapeutic benefit was, in the trial court’s view, fraudulent, because of the lack of “statistical significance” on the primary end point, and the multiple testing with respect to the secondary survival end point.  The Ninth Circuit affirmed the judgment of conviction in an unpublished per curiam opinion[28].

In contrast to the criminal wire fraud prosecution, the civil fraud actions against Dr. Harkonen and the company were dismissed.[29] The prosecution and the judgment in United States v. Harkonen is at odds with the latitude afforded companies in securities fraud cases.  Furthermore, the case cannot be fairly squared with the position that the government took as an amicus curiae in Matrixx Initiatives, Inc. v. Siracusano[30], where the Solicitor General’s office, along with counsel for the Food and Drug Division of the Department of Health & Human Services, in their zeal to assist plaintiffs on claims against an over-the-counter pharmaceutical manufacturer, disclaimed the necessity, or even the importance, of statistical significance[31]:

“[w]hile statistical significance provides some indication about the validity of a correlation between a product and a harm, a determination that certain data are not statistically significant … does not refute an inference of causation.”

Suddenly, when prosecuting an unpopular pharmaceutical company executive, the government’s flexibility evaporated. Government duplicity was a much greater problem than statistical multiplicity in Harkonen.[32]

The Misuse of Power in the Courts

A claim that a study has low power is meaningless unless both the alternative hypothesis and the level of significance are included in the statement.  See Sander Greenland, “Nonsignificance Plus High Power Does Not Imply Support Over the Alternative,” 22 Ann. Epidemiol. 364 (2012); Sander Greenland & Charles, Poole, “Problems in common interpretations of statistics in scientific articles, expert reports, and testimony,” 51 Jurimetrics J. 113, 121-22 (2011).

Power can always be assessed as low by selecting an alternative hypothesis sufficiently close to the null. A study, using risk ratios, which has high power against an alternative hypothesis of 2.0, may have very low power against an alternative of 1.1. Because risk ratios greater than two are often used to attribute specific causation, measuring power of a study against an alternative hypothesis of a doubling of risk might well be a reasonable approach in some cases.  For instance, in Miller v. Pfizer, 196 F. Supp. 2d 1062, 1079 (D. Kan. 2002), aff’d, 356 F. 3d 1326 (10th Cir.), cert. denied, 543 U.S. 917 (2004), the trial court’s Rule 706 expert witness calculated the power of a study to exceed 90% probability to detect a doubling of risk in a pooled analysis of suicidality in clinical trial data of an anti-depressant.  Report of John Concato, M.D., 2001 WL 1793169, *9 (D.Kan. 2001). Unless a court was willing to specify the level at which it would find the risk ratio unhelpful or not probative, such as a relative risk greater than two, power analyses of completed studies are not particularly useful.

Plaintiffs’ counsel rightly complain when defendants claim that a study with a statistically “non-significant” risk ratio greater than 1.0 has no probative value. Although random error (or bias and confounding) may account for the increased risk, the risk may be real. If studies consistently show an increased risk, even though all the studies have reported p-values > 5%, meta-analytic approaches may very well help rule out chance as a likely explanation for the increased risk. The complaint that a study, however, is underpowered, without more, does not help plaintiff establish an association; nor does the complaint establish that the study provides no useful information.

The power of a study depends upon several variables, including the size of the alternative hypothesis, the sample size, the expected value and its variance, and the acceptable level of probability for false-positive findings, which level is reflected in the pre-specified p-value, α, at which level the study’s findings would be interpreted as not likely consistent with the null hypothesis. The lower α is set, the lower will be the power of a test or a study, all other things being equal.  Similarly, moving from a two-tailed to a one-tailed test of significance will increase power.  Courts have acknowledged that both Type I and Type II errors, and the corresponding α and β, are important, but they have overlooked that Type II errors are usually less relevant to the litigation process. See, e.g., DeLuca v. Merrell Dow Pharmaceuticals, Inc., 911 F.2d 941, 948 (3d Cir. 1990).  A single study that failed to show a statistically significant difference in the outcome of interest does not support a conclusion that the outcome is not causally related to the exposure under study.  In products liability litigation, the parties are typically not assigned a burden of proving the absence of causation.

In the Avandia litigation, plaintiffs’ key claim is that the medication, an oral anti-diabetic, causes heart attacks, even though none of the several dozen clinical trials found a statistically significant increased risk. Plaintiffs’ expert witnesses argued that all the clinical trials of Avandia were “underpowered,” and thus the failure to find an increased risk was a Type II (false-negative) error that resulted from the small size of the clinical trials. The Avandia MDL court, considering Rule 702 challenges to plaintiffs’ expert witness opinions, accepted this argument:

“If the sample size is too small to adequately assess whether the substance is associated with the outcome of interest, statisticians say that the study lacks the power necessary to test the hypothesis. Plaintiffs’ experts argue, among other points, that the RCTs [randomized controlled trials] upon which GSK relies are all underpowered to study cardiac risks.”

In re Avandia Mktg., Sales Practices & Prods. Liab. Litig., 2011 WL 13576, at *2 (E.D. Pa. 2011) (emphasis in original). The Avandia MDL court failed to realize that the power argument was empty without a specification of an alternative hypothesis. For instance, in one of the larger trials of Avandia, the risk ratio for heart attack was a statistically non-significant 1.14, with a 95% confidence interval that spanned 0.80 to 1.63. P.D. Home, et al., Rosiglitazone Evaluated for Cardiovascular Outcomes in Oral Agent Combination Therapy for Type 2 Diabetes (RECORD), 373 Lancet  2125 (2009). This trial, standing alone, thus had excellent power against an alternative hypothesis that Avandia doubled the risk of heart attacks; such an alternative hypothesis would clearly be rejected based upon the RECORD trial. On the other hand, an alternative hypothesis of 1.2 would not be. The confidence interval, by giving a quantification of random error, conveys results reasonably compatible with the study estimate; the claim of “low power” against an unspecified alternative hypothesis, conveys nothing.

Last year, in a hormone therapy breast cancer case, the Eight Circuit confused power with β, and succumbed to plaintiff’s expert witness’s argument that he was justified in ignoring several large, well-conducted clinical trials and observational studies because they were “underpowered,” without specifying the alternative hypothesis he was using to make his claim:

“Statistical power is ‘the probability of rejecting the null hypothesis in a statistical test when a particular alternative hypothesis happens to be true’. Merriam–Webster Collegiate Dictionary 973 (11th ed. 2003). In other words, it is the probability of observing false negatives. Power analysis can be used to calculate the likelihood of accurately measuring a risk that manifests itself at a given frequency in the general population based on the sample size used in a particular study. Such an analysis is distinguishable from determining which study among several is the most reliable for evaluating whether a correlative or even a causal relationship exists between two variables.”

Kuhn v. Wyeth, Inc., 686 F.3d 618, 622 n.5 (8th Cir. 2012), rev’g, In re Prempro Prods. Liab. Litig., 765 F. Supp. 2d 1113 (W.D. Ark. 2011). The Kuhn court’s formulation, “in other words,” is incorrect.  Power is not the probability of observing false negatives; it is the probability of correctly rejecting the null in favor of a specified alternative hypothesis, at a specified level of significance probability.  The court’s further discussion of “accurately measuring” mischievously confuses one aspect of statistical power concerned with random variability, with study validity.  The 8th Circuit’s opinion never discusses or discloses what alternative hypothesis the plaintiff’s expert witness had in mind when disavowing certain studies as underpowered.  I suspect that none was ever provided, and that the judges missed the significance of the omission.  The courts would seem better off using the confidence intervals around point estimates to assess the statistical imprecision in the observed data, rather than improper power analyses that fail to specify a legally significant alternative hypothesis.

Some judges and commentators have characterized all evidence as ultimately “probable,” but other writers have criticized this view as trading on the ambiguities inherent in our ordinary usage of probable to convey an epistemic hedge or uncertainty.  How successful is the probabilistic program in the law?  In the context of assessing causation, many courts have succumbed to the temptation to substitute risk for causation.  Other courts have noticed the difference between a prospective risk and a retrospective factual determination that a risk factor actually participated in bringing about the caused result.  In any event, judicial skepticism about probabilistic evidence, in many contexts, has found its expression in holdings and in dicta of common law courts.  The following is a chronological listing of some pertinent cases that rejected or limited the use of overtly probabilistic evidence. There are only two cases involving epidemiological evidence before 1970 on the list.

Day v. Boston & Maine R.R., 96 Me. 207, 217–218, 52 A. 771, 774 (1902) (“Quantitative probability, however, is only the greater chance. It is not proof, nor even probative evidence, of the proposition to be proved. That in one throw of dice, there is a quantitative probability, or greater chance, that a less number of spots than sixes will fall uppermost is no evidence whatever that in a given throw such was the actual result. Without something more, the actual result of the throw would still be utterly unknown. The slightest real evidence would outweigh all the probability otherwise.”)

Toledo, St. L. & W. R. Co. v. Howe, 191 F. 776, 782-83 (6th Cir. 1911) (holding that evidence at issue was not probabilistic, but noting in dictum that “[n]o man’s property should be taken from him on the mere guess that he has committed a wrong. . . because of a probability among other probabilities that the accident for which recovery is sought might have happened in the way charged.”)

People v. Risley, 214 N.Y. 75, 86, 108 N.E. 200, 203 (1915) (holding that probability calculations were improper when “the fact to be established in this case was not the probability of a future event, but whether an occurrence asserted by the people to have happened had actually taken place”)

Lampe v. Franklin Am. Trust, 339 Mo. 361, 384, 96 S.W.2d 710, 723 (1936) (“verdict must be based upon what the jury finds to be facts rather than what they find to be ‘more probable’.”)

Sargent v. Massachusetts Accident Co., 307 Mass. 246, 250, 29 N.E.2d 825, 827 (1940) (the preponderance standard requires more than showing that the chances mathematically favor a fact in dispute; the proponent must prove the proposition in dispute such that the jurors form an actual belief in the truth of the proposition) (“It has been held not enough that mathematically the chances somewhat favor a proposition to be proved; for example, the fact that colored automobiles made in the current year outnumber black ones would not warrant a finding that an undescribed automobile of the current year is colored and not black, nor would the fact that only a minority of men die of cancer warrant a finding that a particular man did not die of cancer. The weight or preponderance of the evidence is its power to convince the tribunal which has the determination of the fact, of the actual truth of the proposition to be proved. After the evidence has been weighed, that proposition is proved by a preponderance of the evidence if it is made to appear more likely or probable in the sense that actual belief in its truth, derived from the evidence, exists in the mind or minds of the tribunal notwithstanding any doubts that may linger there.”)

Smith v. Rapid Transit, 317 Mass. 469, 470, 58 N.E.2d 754, 755 (1945) (evidence that defendant was the only bus franchise operating in the area where the accident took place was not sufficient to establish that the bus that caused the accident belonged to the defendant where private or chartered buses could have been in the area; it is not enough that mathematically the chances somewhat favor the proposition to be proved)

Kamosky v Owens-Illinois Co., 89 F. Supp. 561, 561-62 (M.D.Pa. 1950) (directing verdict in favor of defendant; statistical likelihood that defendant manufactured the bottle that injured plaintiff was insufficient to satisfy plaintiff’s burden of proof)

Mahoney v. United States, 220 F. Supp. 823, 840 41 (E.D. Tenn. 1963) (Taylor, C.J.) (holding that plaintiffs had failed to prove that their cancers were caused by radiation exposures, on the basis of their statistical, epidemiological proofs), aff’d, 339 F.2d 605 (6th Cir. 1964) (per curiam)

In re King, 352 Mass. 488, 491-92, 225 N.E.2d 900, 902 (1967) (physician expert’s opinion that expressed a mathematical likelihood, unsupported by clinical evidence, that claimant’s death from cancer was caused by his accidental fall was legally insufficient to support a judgment)

Garner v. Heckla Mining Co., 19 Utah 2d 367, 431 P.2d 794, 796 97 (1967) (affirming denial of compensation to family of a uranium miner who had smoked cigarettes and had died of lung cancer; statistical evidence of synergistically increased risk of lung cancer among uranium miners is insufficient to show causation of decedent’s lung cancer, especially considering his having smoked cigarettes)

Whitehurst v. Revlon, 307 F. Supp. 918, 920 (E.D. Va. 1969) (holding that challenged evidence was not probabilistic, and noting in dictum that probability evidence of negligence evidence would leave verdict based upon conjecture, guess or speculation)

Guenther v. Armstrong Rubber Co., 406 F.2d 1315, 1318 (3d Cir. 1969) (holding that defendant cannot be found liable on the basis that it supplied 75-80% of the kind of tire purchased by the plaintiff; any verdict based on this evidence “would at best be a guess”)

Crawford v. Industrial Comm’n, 23 Ariz. App. 578, 582-83, 534 P.2d 1077, 1078, 1082-83 (1975) (affirming an employee’s award of no compensation because he was exposed to disease producing conditions both on and off the job; a physician’s testimony, expressed to a reasonable degree of medical certainty that the working conditions statistically increased the probability of developing a disease does not satisfy the reasonable certainty standard)

Olson v. Federal American Partners, 567 P.2d 710, 712 13 (Wyo. 1977) (affirming judgment for employer in compensation proceedings; cigarette smoking claimant failed to show that his lung cancer resulted from workplace exposure to radiation, despite alleged synergism between smoking and radiation).

Heckman v. Federal Press Co., 587 F.2d 612, 617 (3d Cir. 1977) (statistical data about a group do not establish facts about an individual)

Bazemore v. Davis, 394 A.2d 1377, 1382 n.7 (D.C. 1978) (if verdicts were determined on the basis of statistics indicating high probability of alleged facts, more often than not they would be correct guesses, but this is not a sufficient basis for reaching verdicts)

Kaminsky v. Hertz Corp., 94 Mich. App. 356 (1979) (dictum; reversing summary judgment)

Sulesky v. United States, 545 F. Supp. 426, 430 (S.D.W.Va. 1982) (swine flu vaccine GBS cases; epidemiological studies alone do not prove or disprove causation in an individual)

Robinson v. United States, 533 F. Supp. 320, 330 (E.D. Mich. 1982) (finding for government in swine flu vaccine case; the court found that that the epidemiological evidence offered by the plaintiff was not probative, and that it “would reach the same result if the epidemiological data were entirely excluded since statistical evidence cannot establish cause and effect in an individual”)

Iglarsh v. United States, No. 79 C 2148, 1983 U.S. Dist. LEXIS 10950, *10 (N.D.Ill. Dec. 9, 1983) (“In the absence of a statistically valid epidemiological study, even the plaintiff’s treating physician or expert witness, or any clinician for that matter, is unable to attribute a plaintiff’s injury to the swine flu vaccination.”)

Johnston v. United States, 597 F. Supp. 374, 412, 425-26 (D.Kan. 1984) (although the probability of attribution increases with the relative risk, expert must still speculate in making an individual attribution; “a statistical method which shows a greater than 50% probability does not rise to the required level of proof; plaintiffs’ expert witnesses’ reports were “statistical sophistry,” not medical opinion)

Kramer v. Weedhopper of Utah, Inc., 490 N.E.2d 104, 108 (Ill. App. Ct. 1986) (Stamos, J., dissenting) (“Liability is not based on a balancing of probabilities, but on a finding of fact.  While the majority contends that the measure of what is considered sufficient evidence [to support submitting a case to the jury] resolves itself into a question of probability, a review of case law … reveals that a theoretical probability alone cannot be the basis for [a prima facie case].  There must be some evidence in addition to the abstraction which will enable a jury to choose between competing probabilities.”)

Washington v. Armstrong World Industries, 839 F.2d 1121 (5th Cir. 1988) (affirming grant of summary judgment on grounds that statistical correlation between asbestos exposure and disease did not support specific causation)

Thompson v. Merrell Dow Pharm., 229 N.J. Super. 230, 244, 551 A.2d 177, 185 (1988) (epidemiology looks at increased incidences of diseases in populations)

Norman v. National Gypsum Co., 739 F. Supp. 1137, 1138 (E.D. Tenn. 1990) (statistical evidence of risk of lung cancer from asbestos and smoking was insufficient to show individual causation, without evidence of asbestos fibers in the plaintiff’s lung tissue)

Smith v. Ortho Pharmaceutical Corp., 770 F. Supp. 1561, 1576 (N.D. Ga. 1991) (“The court notes that, in an individual case, epidemiology cannot conclusively prove causation; at best, it can establish only a certain probability that a randomly selected case of birth defect was one that would not have occurred absent exposure (or the ‘relative risk’ of the exposed population).”)

Smith v. Ortho Pharmaceutical Corp., 770 F. Supp. 1561, 1573 (N.D. Ga. 1991) (“However, in an individual case, epidemiology cannot conclusively prove causation; at best, it can only establish a certain probability that a randomly selected case of disease was one that would not have occurred absent exposure, or the ‘relative risk’ of the exposed population.  Epidemiology, therefore, involves evidence on causation derived from group-based information, rather than specific conclusions regarding causation in an individual case.”)

Howard v. Wal-Mart Stores, Inc., 160 F.3d 358, 359–60 (7th Cir. 1998) (Posner, C.J.)

Krim v. pcOrder.com, Inc., 402 F.3d 489 (5th Cir. 2005) (rejecting standing plaintiffs’ standing to sue for fraud absent a showing of actual tracing of shares to the offending public offering; statistical likelihood of those shares having been among those purchased was insufficient to confer standing)

United States of America v. W. Scott Harkonen, MD — Part III

Background

The recent oral argument in United States v. Harkonen (see “The (Clinical) Trial by Franz Kafka” (Dec. 11, 2012)), pushed me to revisit the brief filed by the Solicitor General’s office in Matrixx Initiatives Inc. v. Siracusano, 131 S. Ct. 1309 (2011).  One of Dr. Harkonen’s post-trial motions contended that the government’s failure to disclose its Matrixx amicus brief deprived him of a powerful argument that would have resulted from citing the language of the brief, which disparaged the necessity of statistical significance for “demonstrating” causal inferences. See “Multiplicity versus Duplicity – The Harkonen Conviction” (Dec. 11, 2012).

Matrixx Initiatives is a good example of how litigants make bad law when they press for rulings on bad facts.  The Supreme Court ultimately held that pleading and proving causation were not necessary for a securities fraud action that turned on non-disclosure of information about health outcomes among users of the company’s medication. What is required is “materiality,” which may be satisfied upon a much lower showing than causation.  Because Matrixx Initiatives contended that statistical significance was necessary to causation, which in turn was needed to show materiality, much of the briefings before the Supreme Court addressed statistical significance, but the reality is that the Court’s disposition obviated any discussion of the role of statistical inferences for causation. 131 S.Ct. at 1319.

Still, the Supreme Court, in a unanimous opinion, plowed forward and issued its improvident dicta about statistical significance. Taken at face value, the Court’s statement that “the premise that statistical significance is the only reliable indication of causation … is flawed,” is unexceptionable. Matrixx Initiatives, 131 S.Ct. at 1319.  For one thing, the statement would be true if statistical significance were necessary but not sufficient to “indicate” causation. But more to the point, there are some cases in which statistical significance may not be part of the analytical toolkit for reaching a causal conclusion. For instance, the infamous Ferebee case, which did not involve Federal Rule of 702, is a good example of a case that did not involve epidemiologic or statistical evidence.  See “Ferebee Revisited” (Nov. 8, 2012) (discussing the agreement of both parties that statistical evidence was not necessary to resolve general causation because of the acute onset, post-exposure, of an extremely uncommon medical outcome – severe diffuse interstitial pulmonary fibrosis).

Surely, there are other such cases, but in modern products liability law, many causation puzzles are based upon the interpretation of rate-driven processes, measured using epidemiologic studies, involving a measurable base-line risk and an observed higher or lower risk among a sample of an exposed population. In this context, some evaluation of the size of random error is, indeed, necessary. The Supreme Court’s muddled dicta, however, has confused the issues by painting with an extremely broad brush.

The dicta in Matrixx Initiatives has already led to judicial errors. The MDL court in the Chantix litigation provides one such instance. Plaintiffs claimed that Chantix, a medication that helps people stop smoking, causes suicide. Pfizer, the manufacturer, challenged plaintiffs’ general causation expert witnesses, for not meeting the standards of Federal Rule of Evidence 702, for various reasons, not the least of which was that the studies relied upon by plaintiffs’ witnesses did not show statistical significance.  In re Chantix Prods. Liab. Litig., MDL 2092, 2012 U.S. Dist. LEXIS 130144 (Aug. 21, 2012).  The Chantix MDL court, citing Matrixx Initiatives for a blanket rejection of the need to consider random error, denied the defendant’s challenge. Id. at *41-42 (citing Matrixx Initiatives, 131 S.Ct. at 1319).

The Supreme Court, in Matrixx, however, never stated or implied such a blanket rejection of the importance of considering random error in evidence that was essentially statistical in nature. Of course, if it had done so, it would have been wrong.

Within two weeks of the Chantix decision, a similar erroneous interpretation of Matrixx Initiatives surfaced in MDL litigation over fenfluramine.  Cheek v. Wyeth Pharm. Inc., 2012 U.S. Dist. LEXIS 123485 (E.D. Pa. Aug. 30, 2012). Rejecting a Rule 702 challenge to plaintiffs’ expert witness’s opinion, the MDL trial judge, cited Matrixx Initiatives for the assertion that:

“Daubert does not require that an expert opinion regarding causation be based on statistical evidence in order to be reliable. * * * In fact, many courts have recognized that medical professionals often base their opinions on data other than statistical evidence from controlled clinical trials or epidemiological studies.”

Id. at *22 (citing Matrixx Initiatives, 131 S. Ct. at 1319, 1320).  While some causation opinions might be perfectly appropriately based upon other than statistical evidence, the Supreme Court specifically disclaimed any comment upon Rule 702, in Matrixx Initiatives, which was a case about proper pleading of materiality in a securities fraud case, not about proper foundations for actual evidence of causation, at trial, of a health-effects claim. The Cheek decision is thus remarkable for profoundly misunderstanding the Matrixx case. There was no resolution of any Rule 702 issue in Matrixx.

The Trial Court’s Denial of the Matrixx Motion in Harkonen

Dr. Harkonen argued that he is entitled to a new trial on the basis of “newly discovered evidence” in the form of the government’s amicus brief in Matrixx. The trial court denied this motion on several grounds.  First, the government’s amicus brief was filed after the jury returned its verdict against Dr. Harkonen.  Second, the language in the Solicitor General’s amicus brief was just “argument.”  And third, the issue in Matrixx involved adverse events, not efficacy, and the FDA, as well as investors, would be concerned with lesser levels of evidence that did not “demonstrate” causation.  United States v. Harkonen, Memorandum & Order re Defendant Harkonen’s Motions for a New Trial, No. C 08-00164 MHP (N.D. Calif. April 18, 2011). Perhaps the most telling ground might have been that the government’s amicus briefing about statistical significance, prompted by Matrixx Initiatives’ appellate theory, was irrelevant to the proper resolution of that Supreme Court case.  Still, if these reasons are taken individually, or in combination, they fail to mitigate the unfairness of the government’s prosecution of Dr. Harkonen.

The Amicus Brief Behind the Matrixx Motion

Judge Patel’s denial of the motion raised serious problems. See “Multiplicity versus Duplicity – The Harkonen Conviction” (Dec. 11, 2012).  It may thus be worth a closer look at the government’s amicus brief to evaluate Dr. Harkonen’s Matrixx motion. The distinction between efficacy and adverse effects is particularly unconvincing.  Similarly, it does not seem fair to permit the government to take inconsistent positions, whether on facts or on inferences and arguments, when those inconsistencies confuse criminal defendants, prosecutors, civil litigants, and lower court judges. After all, Dr. Harkonen’s use of the key word, “demonstrate” was an argument about the strength and epistemic strength of the evidence at hand.

The government’s amicus brief was filed by the Solicitor General’s office, along with counsel for the Food and Drug Division of the Department of Health & Human Services. The government, in its brief, appeared to disclaim the necessity, or even the importance, of statistical significance:

“[w]hile statistical significance provides some indication about the validity of a correlation between a product and a harm, a determination that certain data are not statistically significant … does not refute an inference of causation.”

Brief for the United States as Amicus Curiae Supporting Respondents, in Matrixx Initiatives, Inc. v. Siracusano, 2010 WL 4624148, at *14 (Nov. 12, 2010). This statement, with its double negatives, is highly problematic.  Validity of a correlation is really not what is at issue in randomized clinical trial; rather it is the statistical reliability or stability of the measurement that is called into question when the result is not statistically significant.  A statistically insignificant result may not refute causation, but it certainly does not thereby support an inference of causation.  The Solicitor General’s brief made this statement without citation to any biostatistics text or treatise.

The government’s amicus brief introduces its discussion of statistical significance with a heading, entitled “Statistical significance is a limited and non-exclusive tool for inferring causation.” Id. at *13.  In a footnote, the government elaborated that its position applied to both safety and efficacy outcomes:

“[t]he same principle applies to studies suggesting that a particular drug is efficacious. A study  in which the cure rate for cancer patients who took a drug was twice the cure rate for those who took a placebo could generate meaningful interest even if the results were not statistically significant.”

Id. at *15 n.2.  Judge Patel’s distinction between efficacy and adverse events thus cannot be sustained. Of course, “meaningful interest” is not exactly a sufficient basis for a causal conclusion. As a general matter, Dr. Harkonen’s motion seems well grounded.  Although not a model of clarity, the amicus brief appears to disparage the necessity of statistical significance for supporting a causal conclusion. A criminal defendant being prosecuted for using the wrong verb to describe his characterization of the inference he drew from a clinical trial would certainly want to showcase these high-profile statements made by Solicitor General’s office to the highest court of the land.

Solicitor General’s Good Advice

Much of the Solicitor General’s brief is directly on point for the Matrixx case. The amicus brief leads off by insisting that information that supports reasonable suspicions about adverse events, may be material absent sufficient evidence of causation.  Id. at 11.  Of course, this is the dispositive argument, and it is stated well in the brief.  The brief then wonders into scientific and statistical territory, with little or no authority, at times misciting important works such as the Reference Manual on Scientific Evidence.

The Solicitor General’s amicus brief hones in on the key issue: materiality, which does not necessarily involve causation:

“Second, a reasonable investor may consider information suggesting an adverse drug effect important even if it does not prove that the drug causes the effect.”

Brief for the United States as Amicus Curiae Supporting Respondents, in Matrixx Initiatives, Inc. v. Siracusano, 2010 WL 4624148, at *8.

“As explained above (see p. 19, supra), however, adverse event reports do not lend themselves to a statistical-significance analysis. At a minimum, the standard petitioners advocate would require the design of a scientific study able to capture the relative rates of incidence (either through a clinical trial or observational study); enough participants and data to perform such a study and make it powerful enough to detect any increased incidence of the adverse effect; and a researcher equipped and interested enough to conduct it.”

Id. at 23.

“As petitioners acknowledge (Br. 23), FDA does not apply any single metric for determining when additional inquiry or action is necessary, and it certainly does not insist upon ‘statistical significance.’ See Adverse Event Reporting 7. Indeed, statistical significance is not a scientifically appropriate or meaningful standard in evaluating adverse event data outside of carefully designed studies. Id. at 5; cf. Lempert 240 (‘it is meaningless to talk about receiving a statistically significant number of complaints’).”

Id. at 19. So statistical significance is unrelated to the case, and the kind of evidence of materiality, alleged by plaintiffs, does not even open itself to a measurement of statistical significance.  At this point, the brief writers might have called it a day.  The amicus brief, however, pushes on.

Solicitor General’s Ignoratio Elenchi

A good part of the government’s amicus brief in Matrixx presented argument irrelevant to the issues before the Court, even assuming that statistical significance was relevant to materiality.

“First, data showing a statistically significant association are not essential to establish a link between use of a drug and an adverse effect. As petitioners ultimately acknowledge (Br. 44 n.22), medical researchers, regulators, and courts consider multiple factors in assessing causation.”

Brief for the United States as Amicus Curiae Supporting Respondents, in Matrixx Initiatives, Inc. v. Siracusano, 2010 WL 4624148, at *12.  This statement is a non-sequitur.  The consideration of multiple factors in assessing causation does not make the need for a statistically significant association or more less essential. Statistical significance could still be necessary but not sufficient in assessing causation.  The government’s brief writers pick up the thread a few pages later:

“More broadly, causation can appropriately be inferred through consideration of multiple factors independent of statistical significance. In a footnote, petitioners acknowledge that critical fact: ‘[C]ourts permit an inference of causation on the basis of scientifically reliable evidence other than statistically significant epidemiological data. In such cases experts rely on a lengthy list of factors to draw reliable inferences, including, for example,

(1) the “strength” of the association, including “whether it is statistically significant”;

(2) temporal relationship between exposure and the adverse event;

(3) consistency across multiple studies;

(4) “biological plausibility”;

(5) “consideration of alternative explanations” (i.e., confounding);

(6) “specificity” (i.e., whether the specific chemical is associated with the specific disease at issue); and

(7) dose-response relationship (i.e., whether an increase in exposure yields an increase in risk).’ ”

Pet. Br. 44 n.22 (citations omitted). Those and other factors for inferring causation have been well recognized in the medical literature and by the courts of appeals. See, e.g., Reference Guide on Epidemiology 345-347 (discussing relevance of toxicologic studies), 375-379 (citing, e.g., Austin Bradford Hill, The Environment and Disease: Association or Causation?, 58 Proc. Royal Soc’y Med. 295 (1965))… .”

Id. at 15-16. These enumerated factors are obviously due to Sir Austin Bradford Hill. No doubt Matrixx Initiatives cited the Bradford Hill factors, but that was because the company was contending that statistical significance was necessary but not sufficient to show causation.  As Bradford Hill showed by his famous conclusion that smoking causes lung cancer, these factors were considered after statistical significance was shown in several epidemiologic studies.  The Supreme Court incorporated this non-argument into its opinion, even after disclaiming that causation was needed for materiality or that the Court was going to assess the propriety of causal findings in other cases.

The Solicitor General went on to cite three cases for the proposition that statistical significance is not necessary for assessing causation:

“Best v. Lowe’s Home Centers, Inc., 563 F.3d 171, 178 (6th Cir. 2009) (“an ‘overwhelming majority of the courts of appeals’ agree” that differential diagnosis, a process for medical diagnosis that does not entail statistical significance tests, informs causation) (quoting Westberry v. Gislaved Gummi AB, 178 F.3d 257, 263 (4th Cir. 1999)).”

Id. at 16.  These two cases both involved so-called “differential diagnosis” or differential etiology, a process of ruling in, by ruling out.  This method, which involves iterative disjunctive syllogism, starts from established causes, and reasons to a single cause responsible for a given case of the disease.  The citation of these cases was irrelevant and bad scholarship by the government.  The Solicitor General’s error here seems to have been responsible for the Supreme Court’s unthinking incorporation of these cases into its opinion.

The Solicitor General went on to cite a third case, the infamous Ferebee, for its suggestion that statistical significance was not necessary to establish causation:

“Ferebee v. Chevron Chem. Co., 736 F.2d 1529, 1536 (D.C. Cir.) (‘[P]roducts liability law does not preclude recovery until a “statistically significant” number of people have been injured’.), cert. denied, 469 U.S. 1062 (1984). As discussed below (see pp. 19-20, infra), FDA relies on a number of those factors in deciding whether to take regulatory action based on reports of an adverse drug effect.”

Id. at 16.  Curiously, the Supreme Court departed from its reliance on the Solicitor General’s brief, with respect to Ferebee, and substituted its own citation to Wells v. Ortho Pharmaceutical Corp., 615 F. Supp. 262 (N.D. Ga. 1985), aff’d in relevant part, 788 F.2d 741 (11^th Cir.), cert. denied, 479 U.S.950 (1986). See Wells v. Ortho Pharmaceutical Corp. Reconsidered – Part 1 (Nov. 12, 2012).  The reliance upon the two differential etiology cases was “demonstrably” wrong, but citing Wells was even more bizarre because that case featured at least one statistically significant study relied upon by plaintiffs’ expert witnesses. Ferebee, on the other hand, involved an acute onset of a rare condition – severe pulmonary fibrosis – shortly after exposure to paraquat.  Ferebee was thus a case in which the parties agreed that the causal relationship between paraquat and lung fibrosis had been established by non-analytical epidemiologic evidence.  See Ferebee Revisited.

The government then pointed out in its amicus that sometimes statistical significance is hard to obtain:

“In some circumstances —e.g., where an adverse effect is subtle or has a low rate of incidence —an inability to obtain a data set of appropriate quality or quantity may preclude a finding of statistical significance. Ibid. That does not mean, however, that researchers have no basis on which to infer a plausible causal link between a drug and an adverse effect.”

Id. at 15. Biological plausibility is hardly a biologically established causal link.  Inability to find an appropriate data set often translates into an inability to draw a causal conclusion; inappropriate data are not an excuse for jumping to unsupported conclusions.

Solicitor General’s Bad Advice – Crimen Falsi?

The government’s brief then manages to go from bad to worse. The government’s amicus brief in Matrixx raises serious concerns about criminalizing inappropriate statistical statements, inferences, or conclusions.  If the Solicitor General’s office, with input from Chief Counsel of the Food and Drug Division, of the Department of Health & Human Services, cannot correctly state basic definitions of statistical significance, then the government has no business of prosecuting others for similar offenses.

“To assess statistical significance in the medical context, a researcher begins with the ‘null hypothesis’, i.e., that there is no relationship between the drug and the adverse effect. The researcher calculates a ‘p-value’, which is the probability that the association observed in the study would have occurred even if there were in fact no link between the drug and the adverse effect. If that p-value is lower than the ‘significance level’ selected for the study, then the results can be deemed statistically significant.”

Id. at 13. Here the government’s brief commits a common error that results when lawyers want to simplify the definition of a p-value. The p-value is a cumulative probability of observing a disparity at least as great as observed, given the assumption that there is no difference.  Furthermore, the subjunctive is not appropriate to describe the basic assumption of significance probability.

“The significance level most commonly used in medical studies is 0.05. If the p-value is less than 0.05, there is less than a 5% chance that the observed association between the drug and the effect would have occurred randomly, and the results from such a study are deemed statistically significant. Conversely, if the p-value is greater than 0.05, there is greater than a 5% chance that the observed association would have occurred randomly, and the results are deemed not statistically significant. See Reference Guide on Epidemiology 357-358; David Kaye & David A. Freedman, Reference Guide on Statistics, in Reference Manual on Scientific Evidence 123, 123-125 (2d ed. 2000) (Reference Guide on Statistics).”

Id. at 14. Here the government’s brief drops the conditional of the significance probability; the p-value provides the probability that a disparity at least as large as observed would have occurred (based upon the assumed probability model), given the assumption that there really is no difference between the observed and expected results.

“While statistical significance provides some indication about the validity of a correlation between a product and a harm, a determination that certain data are not statistically significant – let alone, as here, the absence of any determination one way or the other — does not refute an inference of causation. See Michael D. Green, Expert Witnesses and Sufficiency of Evidence in Toxic Substances Litigation: The Legacy of Agent Orange and Bendectin Litigation, 86 Nw. U. L. Rev. 643, 682- 683 (1992).”

Id. at 14. Validity is probably the wrong word since most statisticians and scientific authors use validity to refer to features other than low random error.

“Take, for example, results from a study, with a p-value of 0.06, showing that those who take a drug develop a rare but serious adverse effect (e.g., permanent paralysis) three times as often as those who do not. Because the p-value exceeds 5%, the study’s results would not be considered statistically significant at the 0.05 level. But since the results indicate a 94% likelihood that the observed association between the drug and the effect would not have occurred randomly, the data would clearly bear on the drug’s safety. Upon release of such a study, “confidence in the safety of the drug in question should diminish, and if the drug were important enough to [the issuer’s] balance sheet, the price of its stock would be expected to decline.” Lempert 239.²”

Id. at 14-15. The citation to Lempert’s article is misleading. At the cited page, Professor Lempert is simply making the point that materiality in a securities fraud case will often be present when evidence for a causal conclusion is not. Richard Lempert, “The Significance of Statistical Significance:  Two Authors Restate An Incontrovertible Caution. Why A Book?” 34 Law & Social Inquiry 225, 239 (2009).  In so writing, Lempert anticipated the true holding of Matrixx Initiative.  The calculation of the 94% likelihood is also incorrect.  The quantity (1 – [p-value]) yields a probability that describes the probability of obtaining a disparity no greater than the observed result, on the assumption that there is no difference at all between observed and expect results. There is, however, a larger point lurking in this passage of the amicus brief, which is the difference between a p-value of 0.05 and 0.06 is not particularly large, and there is thus a degree of arbitrariness to treating it as too sharp a line.

All in all, a distressingly poor performance by the Solicitor General’s office.  With access to many talented statisticians, the government could have at least have had a competent statistician review and approve the content of this amicus brief.  I suspect that most judges and lawyers, however, would balk at drawing an inference that the Solicitor General intended to mislead the Court simply because the brief contained so many misstatements about statistical inference.  This reluctance should have obvious implications for the government’s attempt to criminalize Dr. Harkonen’s statistical inferences.