TORTINI

For your delectation and delight, desultory dicta on the law of delicts.

Archives »

The Role of the Material Safety Data Sheet in Expert Witness Gatekeeping

June 23rd, 2012

Several years ago, I had an unusual workman’s compensation case, pending in Salem County, New Jersey.  The petitioner claimed that after his employment, he had developed a rare disease as a result of his workplace exposures.  The exact disease really does not matter, other than to note that there was some suggestive epidemiologic evidence of an association, and some evidence against the association.  As a result of the scientific ambiguity, the respondent had started to warn, on its material data safety sheets, and on its packaging, of the association.  So when I stepped into the judge’s chambers, the first thing the petitioner’s counsel said was:  “I win; the employer has admitted causation.”  The judge looked skeptical, and when I pointed out that association is not causation, and that warnings are not statements of scientific conclusions, His Honor agreed enthusiastically.  To the petitioner’s counsel’s shock and dismay, the judge directed us to brief the reliability issue.  Things like that do not often happen in New Jersey, and especially not in a worker’s compensation case.  (A statute of limitations issue ultimately turned out to be dispositive, and we never did get a ruling on the scientific claim.)

The use of the material safety data sheet (MSDS), or warnings on product packaging, is a recurring theme in so-called toxic tort litigation.  The MSDS is a compilation of information about a hazardous material or substance.  Under Occupational Safety and Health Administration (OSHA) regulations, the seller is required to compile information about hazards and provide to purchasers. See 29 C.F.R. § 1910.1200(g) (2006).  Much of the required information is regulatory classification, which is often based upon precautionary judgments about hazards and risks, without information specific to actual exposures likely experienced by end users.  The existence of the MSDS often provides claimants a basis to argue that the seller has admitted causation, but in reality, the argument is little more than a substitution of substituting regulatory precautionary judgment for causal assessments.

The Fifth Circuit’s recent, sure-footed decision in Johnson v. Arkema, Inc., below, reminded me how misleading and how persistent are the expert witnesses who argue causal conclusions based upon MSDS.  The Fifth Circuit recognized that an MSDS cannot be more reliable than the evidence upon which it is based.  Courts need to be on guard against the seductive argument that warnings and MSDS should substitute for scientific evidence of causation.  Unfortunately, not all judges are as astute as the judge I drew in my Salem County worker’s compensation case, which means that there is work to be done.

 

FEDERAL CASES REJECTING RELIANCE UPON MSDS

Johnson v. Arkema Inc., Slip op. at 11-12, 2012 WL ___ (5th Cir. June 20, 2012) (per curiam) (affirming exclusion of expert witnesses who relied in part upon MSDS for two chemicals when the MSDS identified only a very general physical reaction of “respiratory tract irritation,” without identifying the underlying scientific support or specifying the relevant duration and exposure needed to induce any particular adverse outcome)

Pritchard v. Dow AgroSciences, LLC, 705 F. Supp. 2d 471 (W.D. Pa. 2010) (excluding expert witness who opined that Dursban caused NHL based in part upon MSDS), aff’d, 430 Fed. Appx. 102 (3d Cir. 2011), cert. denied, 132 S. Ct. 508 (2011)

Seaman v. Seacor Marine LLC, 564 F. Supp. 2d 598, 603 (E.D. La. 2008) (rejecting reliance on MSDS because, inter alia, “the MSDS … does not mention bladder cancer … as a potential effect”), aff’d, 326 F. App’x 721, 726 (5th Cir. 2009)

Turner v. Iowa Fire Equip. Co., 229 F.3d 1202, 1209 (8th Cir. 2000) (affirming exclusion of expert witness who relied upon MSDS among other things)

Mitchell v. Gencorp Inc., 165 F.3d 778, 781 (10th Cir. 1999) (reliance upon MSDS insufficient)

Moore v. Ashland Chem. Inc., 151 F.3d 269, 278 (5th Cir. 1998) (en banc) (holding that the district court did not abuse its discretion in finding an expert witness’s reliance upon an MSDS because the withness “did not know what tests Dow [Corning] had conducted in generating the MSDS”)

Leake v. United States, 2011 U.S. Dist. LEXIS 149634 (E.D. Pa. 2011) (excluding expert witness who concluded that painting exposure caused liver failure based upon MSDS, etc.)

Henricksen v. ConocoPhillips Co., 605 F. Supp. 2d 1142, 1159 (E.D. Wash. 2009)(excluding expert witness who relied upon MSDS, among other items)

Moore v. P&G-Clairol,  Case No. 09 C 1723, Slip op. (N.D. Ill. March 18, 2011)(excluding expert witness who relied upon MSDS to support a claim of a severe allergic reaction to Clairol hair dye; the MSDS did not address human reactions or consumer exposure levels)

STATE DECISIONS REJECTING RELIANCE UPON MSDS

OHIO

Braglin v. Lempco Indus., Inc., 2007 Ohio 1964; 2007 Ohio App. LEXIS 1773 (2007) (affirming exclusion of expert witness who relied upon MSDS among other things, and opined that plaintiff’s pancreatic cancer was caused by chemicals in raw metal processing)

TEXAS

Brookshire Bros., Inc. v. Smith, 176 S.W.3d 37-38 & n.7, 2003 WL 21756411, *4 (Tex. App. – Houston [1st Dist.] 2003, no pet. ) (MSDS or warning label cannot, alone, provide the specific, detailed, reliable showing of causation to support an expert witness’s opinion)

Coastal Tankships U.S.A. Inc. v. Anderson, 87 S.W.3d 591, 611 (Tex. App.‑ Houston [1st Dist.] 2002, pet. denied) (statement of health “effects” in MSDS was not reliable evidence of causation)

See also Exxon Corp. v. Makofski, 116 S.W.3d 176, 187-88 (Tex. App. 2003) (“standards used by OSHA [and] the EPA” inadequate for causal determinations)

 

UNTOWARD DECISIONS

There is, of course, some factual complexity to these decisions; some MSDS obviously will list well-established causal relationship; others not.  The key point is that an MSDS is a tertiary source, compiled from primary sources at various levels in the hierarchy of evidence, as well as secondary reviews.  To make matters really murky, most MSDS must also report regulatory classifications and determinations, which often are not evidence-based.  Most of the “untoward” decisions, below, share typical fallacious elements:

  • treating MSDS as an admission by the manufacturer or seller;
  • confusing precautionary regulatory assessments with scientific determinations;
  • ignoring considerations of dose, exposure, route of exposure, animal species; and
  • confusing disease outcome discussed in MSDS with that claimed by plaintiff.

FEDERAL

Best v. Lowe’s Home Centers, Inc., 563 F.3d 171 (6th Cir. 2009), rev’g No. 3:04-CV-294, 2008 WL 2359986 (E.D. Tenn. June 5, 2008) (excluding expert witness who relied extensively upon MSDS)

Curtis v.M&S Petroleum, Inc. 174 F.3d 661, 669-70 (5th Cir. 1999) (reliance upon a particular MSDS was reasonable when the sheet was consistent with a body of reliable information about the hazards of benzene exposure)

Westberry v. Gislaved Gummi AB, 178 F.3d 257, 264-66 (4th Cir. 1999) (expert witness’s causation opinion of plaintiff’s sinus condition was reasonably based upon facts and data, including MSDS on talc)

McCullock v. H.B. Fuller Co., 61 F.3d 1038, 1043-44 (2d Cir.1995) (affirming denial of Rule 702 motion when expert witness’s causation opinion was based upon wide array of materials, including product’s MSDS)

Allen v. Martin Surfacing, 263 F.R.D. 47 (D. Mass. 2009) (denying challenge to expert witness who concluded that plaintiff’s exposure to neurotoxic levels of floor-surfacing chemical caused his ALS)

In re Stand ‘n Seal Prod. Liab. Litig., 1:07 MD1804-TWT, MDL 1804, Order Sur Longo (N.D. Ga. June 15, 2009)(denying motion to exclude expert witnesses who relied in part upon MSDS)

In re Welding Fume Prod. Liab. Litig., 2006 WL 4507859, *35 (N.D.Ohio 2006)( OMALLEY, J.); 2005 WL 1868046, *36 (N.D.Ohio) (denying challenge to expert witnesses who claimed that welding causes Parkinson’s disease on basis of general statements in MSDS that a component of welding fume (manganese) can be neurotoxic, without specification of dose or duration)

Westley v Ecolab, Inc., 2004 WL 1068805 (E.D.Pa. 2004) (denying motion against expert witness who relied upon MSDS)

Blandin Paper Co. v. J&J Industrial Sales, Inc., No. Civ.02-4858 ADM/RLE, 2004 WL 1946388 (D. Minn. Sept. 2, 2004) (denying motion to exclude plaintiff’s expert witness who relied in part upon MSDS)

Lentz v. Mason, Case No. 1:96-cv-02319-SMO, Slip op. (D.N.J. Jan. 11, 1999)(denying motion to exclude expert witness who relied up MSDS as well as independent testing)

STATE

Langness v. Fencil Urethane Systems, Inc., 2003 ND 132, 667 N.W.2d 596 (2003) (reversing jury verdict on grounds of error in excluding expert witness who relied in part upon MSDS for physical properties of material)

Posted in Causation, Rule 702 | Comments Off on The Role of the Material Safety Data Sheet in Expert Witness Gatekeeping

Johnson v. Arkema Inc. – The Fifth Circuit Proves to Be Sophisticated Consumer of Science

June 21st, 2012

Yesterday, in celebration of the first day of summer, the Fifth Circuit handed down a decision in a case that looks like a laundry list of expert witness fallacies.  Fortunately, the district judge and two of the three appellate judges kept their analytical faculties intact.  Johnson v. Arkema Inc., Slip op., 2012 WL ___ (5th Cir. June 20, 2012) (per curiam) (affirming exclusion of expert witnesses).

The plaintiff had worked in a glass bottling plant, where on two occasions in 2007, he was in close proximity to the defendant’s ventilation hood, designed to be used with a chemical, Certincoat, composed of monobutyltin trichloride (MBTC), an organometallic compound.  Plaintiff claimed that the ventilation was inadequate and that as a result he was exposed to MBTC as well as hydrochloric acid.

The plaintiff sustained some acute symptoms and ultimately was diagnosed with a “chemical pneumonia,” by his treating physician.  The plaintiff further claimed that his condition progressively worsened,  and that he was ultimately diagnosed with “pulmonary fibrosis,” a “severe restrictive lung disease.” The plaintiff filed reports from two expert witnesses – Richard Schlesinger, a toxicologist, and Charles Grodzin, a pulmonary physician – in support of his claim that his pulmonary fibrosis was caused by overexposure to MBTC and hydrochloric acid (HCl).

Plaintiff’s claim led to defendant’s Rule 702 challenge, which the trial court sustained, and the appellate court affirmed.

A basic problem faced by plaintiff is that there was virtually no evidence that MBTC or HCl causes pulmonary fibrosis. Undaunted, the plaintiff and his expert witnesses pushed on, but the lack of epidemiologic evidence associating MBTC or HCl with pulmonary fibrosis proved reliably harmful to plaintiff’s case.

General Acceptance

Plaintiff could point to no evidence that MBTC or HCl causes pulmonary fibrosis.  Slip op. at 7. Given the delay in manifestation of the fibrosis after the plaintiff’s rather limited, discrete exposures, the court recognized that epidemiologic evidence was important, if not essential, to plaintiff’s case. Without epidemiology, the plaintiff retreated to generalities – the chemicals cause lung irritation, lung injury, etc.  One concurring judge was taken in, but the majority of the panel saw through the dodge.

Anecdotal Evidence

Without epidemiologic evidence, the plaintiff invoked anecdotal evidence that other employees sustained similar lung injuries. The problem, however, for even this low-level evidence was that other employees experienced only transitory symptoms, which quickly resolved.  Id. at 4 -5, 27.

Post Hoc, Ergo Propter Hoc

Focusing only on himself as an anecdote with n =1, the plaintiff, and his expert witnesses, argued that temporal sequence of his exposure and his pulmonary fibrosis was itself evidence of causality.  Neither the trial court nor the appellate court found this much of an argument.  Id. at 16 n.13, 18.

Mechanism in Search of Data – Schlesinger’s irritant theory

Schlesinger argued that both MBTC and HCl are pulmonary irritants, which can cause inflammation, and pulmonary fibrosis results from inflammation. Id. at 8.  True, but not all irritants cause pulmonary fibrosis.  Chronicity and dose are important considerations.  Whether these chemicals, under exposure conditions experienced by plaintiff, were capable of causing pulmonary fibrosis, cried out for evidence.

The Material Safety Data Sheets (MSDS)

The plaintiff argued that the MSDS for HCl established that this chemical was “severely corrosive to the respiratory system.” Id. at 11-12.  The defendant’s own MSDS for MBTC stated that MBTC “causes respiratory tract irritation.” Id. at 16.  The courts saw these arguments as transparently absent evidence. None of the MSDS identified pulmonary fibrosis; nor did they specify (1) the underlying scientific support, or (2) the relevant duration and exposure needed to induce any particular adverse outcome.

Animal Studies

For both MBTC and HCl, plaintiff adverted to animal studies, but the courts found that the animal studies failed to support the plaintiff’s expert witnesses’ opinions and the plaintiff’s claims.  The studies were readily distinguishable in terms of dose, duration, and disease outcome.  In particular, none of the studies showed that the chemicals caused pulmonary fibrosis. Id. at 7, 12 (baboon study of HCl showed impairment but not fibrosis at 10,000ppm for one year, quite unlike plaintiff’s exposure), 16-17 (rat inhalation study of MBTC, six hrs/day, five days/wk, up to 30 mg/m3, with toxicity but no mention of lung fibrosis).

Regulatory Limits

Plaintiff argued that HCl levels were multiples of the OSHA limits, but the courts would not credit regulatory exposure limits are evidence of harmfulness because of the precautionary nature of many regulations.  Id. at 14.  Furthermore, the disease outcomes of regulatory concern did not appear to be pulmonary fibrosis for the chemicals involved.

Res Ipsa Loquitur

The plaintiff argued that causation was a matter of common sense and general experience.  Even if his expert witnesses did not have valid, reliable evidence, the jury could make the causal determination without scientific evidence. Id. at  26.  Rejected.

Chemical Analogies

The defendant’s expert witness acknowledged that tin oxide can cause pulmonary fibrosis.  Id. at 28.  This admission, however, came without any qualification about what exposure or duration data might be needed to support a conclusion about specific causation in the plaintiff.  Id.  Furthermore, tin pneumoconiosis, or stannosis, is known as a benign lung disease, unassociated with impairment or disability.  Like simple silicosis, stannosis is a picture change on chest radiograph, without diminution of performance on pulmonary function tests.  Agency for Toxic Substances and Disease Registry, A Toxicological Profile for Tin and Tin Compounds at 30 (2005).

Differential Diagnosis

Plaintiff’s pulmonary expert witness, Dr. Grodzin, tried to bootstrap specific causation by assuming general and putting it in the “differentials” for him to embrace.  Id. at 19.  A fallacious form of reasoning, but the courts here were on top of it.

* * * * *

The panel did reverse the trial court’s grant of summary judgment.  The gate closed a little too fast to permit scrutiny of plaintiff’s claim of acute injuries and symptoms, which were less dependent upon epidemiologic evidence.

 

Posted in Causation, Expert Witnesses, Rule 702, Scientific Evidence | Comments Off on Johnson v. Arkema Inc. – The Fifth Circuit Proves to Be Sophisticated Consumer of Science

Meta-Meta-Analysis – Celebrex Litigation – The Claims – Part One

June 21st, 2012

In the Celebrex/Bextra litigation, both sides acknowledged the general acceptance and validity of meta-analysis, for both observational studies and clinical trials, but attacked the other side’s witnesses’ meta-analyses on grounds specific to how they were conducted.  See, e.g., Pfizer Defendants’ Motion to Exclude Certain Plaintiffs’ Experts’ Causation Opinion Regarding Celebrex – Memorandum of Points and Authorities in Support Thereof at 14, 16 (describing meta-analysis as “appropriate” and a “useful way to evaluate the presence and consistency of an effect,” and “a valid technique for analyzing the results of both randomized clinical trials and observational studies”)(dated July 20, 2007), submitted in MDL 1699, In re Bextra and Celebrex Marketing Sales Practices & Prod. Liab. Litig., Case No. 05-CV-01699 CRB (N.D. Calif.) [hereafter MDL 1699]; Plaintiffs’ Memorandum of Law in Support of Their Motion to Exclude Expert Testimony by Defendants’ Expert Dr. Lee-Jen Wei at 2 (July 23, 2009) (“While use of a properly conducted meta-analysis is appropriate, there are underlying scientific principles and techniques to be used in meta-analysis that are widely accepted among biostatisticians and epidemiologists. Wei’s meta-analysis – which he acknowledges is based in part on an admittedly novel approach that is not generally recognized by the scientific community – fails to follow certain of these key principles.”), submitted in In re Pfizer, Inc. Securities Litig., Nos. 04 Civ. 9866(LTS)(JLC), 05 md 1688(LTS) (S.D.N.Y.)[hereafter Securities Litig.]

The plaintiffs and defendants expended a great deal of energy in attacking the other side’s meta-analyses as conducted.  With all the briefing in the federal MDL, the New York state cases, and the securities fraud class action, hundreds of pages were written on the suspected flaws in meta-analyses.  The courts, in both the products liability MDL cases and in the securities case, denied the challenges in a few sentences.  Indeed, it is difficult if not impossible to discern what the challenges were from reading the courts’ decisions. In re Pfizer Inc. Securities Litig., 2010 WL 1047618 (S.D.N.Y. 2010); In re Bextra and Celebrex, 2008 N.Y. Misc. LEXIS 720; 239 N.Y.L.J. 27(2008); In re Bextra and Celebrex Marketing Sales Practices and Product Liability Litig., MDL No. 1699, 524 F.Supp. 2d 1166 (N.D. Calif. 2007)

Although the issues shifted some over the course of these litigations, certain important themes recurred.  The plaintiffs focused their attack upon the meta-analyses conducted by defense expert witness, Lee-Jen Wei, a professor of biostatistics at the Harvard School of Public Health.

The plaintiffs maintained that Professor Wei’s meta-analyses should be excluded under Rule 702, or the New York case law, because of

  • inclusion of short-term clinical trials
  • failure to weight risk ratios by person years
  • inclusion of zero-event trials with use of imputation methods
  • use of risk difference instead of risk ratios
  • use of exact confidence intervals instead of estimated intervals

See generally Plaintiffs’ Memorandum of Law in Support of Their Motion to Exclude Expert Testimony by Defendants’ Expert Dr. Lee-Jen Wei (July 23, 2009), in Securities Litig.

The plaintiffs advanced meta-analyses conducted by Professor David Madigan, Professor and Chair in the Department of Statistics, Columbia University.  The essence of the defendants’ challenges revolved around claims of flawed outcome and endpoint ascertainment and definitions:

  • invalid clinical endpoints
  • flawed data collection procedures
  • ad hoc changes in procedure and methods
  • novel methodologies “never used in the history of clinical research”
  • lack of documentation for classifying events
  • absence of expert clinical judgment in classifying event for inclusion in meta-analysis
  • creation of composite endpoints that included events unrelated to plaintiffs’ theory of thrombotic mechanism
  • lack of blinding to medication use when categorizing events
  • failure to adjust for multiple comparisons in meta-analyses

See generally Pfizer Defendants’ Motion to Exclude Certain Plaintiffs’ Experts’ Causation Opinion Regarding Celebrex – Memorandum of Points and Authorities in Support Thereof (dated July 20, 2007), in MDL 1699; Pfizer defendants’ [Proposed] Findings of Fact and Conclusions of Law with Respect to Motion to Exclude Certain Plaintiffs’ Experts’ Opinions Regarding Celebrex and Bextra, and Plaintiffs’ Motion to Exclude Defendants’ Expert Dr. Lee-Jen Wei, Document 175, submitted in Securities Litig. (Dec. 4, 2009).

Why did the three judges involved (Judge Breyer in the federal MDL; Justice Kornreich in the New York state cases; and Judge Swain in the federal securities putative class action) give such cursory attention to these Rule 702/Frye challenges?  The complexity of the issues, the lack of clarity in the lawyers’ briefings, and the stridency of both sides perhaps contributed to shorten judicial attention span.  Some of the claims were simply untenable, and may have obliterated more telling critiques.

ZERO-EVENT TRIALS

Many of the Celebrex parties’ claims can be traced to a broader issue of what to include or exclude in a meta-analysis.  Consider for instance the plaintiffs’ challenge to Wei’s meta-analysis.  The plaintiffs faulted Wei for including short-term clinical trials in his meta-analysis, while sponsoring their own expert witness testimony that Celebrex could induce heart attack or stroke after first ingestion of the medication.  Having made the claim, the plaintiffs were hard pressed to exclude short-term trials, other than to argue that such trials frequently had zero adverse events in either the medication or placebo arms.  Many meta-analytic methods, which treat each included study as a 2 x 2 contingency table, and calculate an odds ratio for each table, cannot accommodate zero event data.

Whether or not hard pressed, the plaintiffs made the claim. The plaintiffs’ analogized to the lack of reliability of underpowered clinical trials to provide evidence of safety.  See Plaintiffs’ Reply Memorandum of Law in Further Support of Their Motion to Exclude Expert Testimony by Defendants’ Expert Dr. Lee-Jen Wei at 6 (May 5, 2010), in Securities Litig. (citing In re Neurontin Mktg., Sales Practices, and Prod. Liab. Litig., 612 F. Supp. 2d 116, 141 (D. Mass. 2009) (noting that many of Pfizer’s studies were “underpowered” to detect the alleged connection between Neurontin and suicide).  The power argument, however, does not make sense in the context of a meta-analysis, which is aggregating data across studies to overcome the alleged lack of power in a single study.

Not surprisingly, clinical trials of a non-cardiac medication will often report no event of the outcome of interest, such as heart attack.  These trials are referred to as a “zero event”, which can happen in one or both arms of a given trial.  Some searchers exclude these studies from a meta-analysis because of the impossibility of calculating an odds ratio without using imputation in the zero cells of the 2 x 2 tables. Although there are methods to address zero-event trials, some researchers believe that the existence of several zero-event trials essentially means that the sparse data from rare outcomes deprives statistical tests of their usual meaning.  Traditional statistical standards of significance (p < 0.05) are described as “tenuous,” and too high, in this situation. A.V. Hernandez, E. Walker, J. P. Ioannidis, M.W. Kattan, “Challenges in meta-analysis of randomized clinical trials for rare harmful cardiovascular events: the case of rosiglitazone,” 156 Am. Heart J. 23, 28 (2008).

The exclusion of zero-event trials from meta-analyses of rare outcomes can yield biased results. See generally M.J. Bradburn, J.J Deeks, J.A. Berlin, and A. Russell Localio,” Much ado about nothing: a comparison of the performance of meta-analytical methods with rare events,” 26 Statistics in Med. 53 (2007); M.J. Sweeting, A.J. Sutton, and P.C. Lambert, “What to add to nothing? Use and avoidance of continuity corrections in meta-analysis of sparse data,” 23 Statistics in Med. 1351 (2004)(erratum at 25 Statistics in Med. 2700 (2006) (“Many routinely used summary methods provide widely ranging estimates when applied to sparse data with high imbalance between the size of the studies’ arms. A sensitivity analysis using several methods and continuity correction factors is advocated for routine practice.”).

Others researchers include zero-event trials as providing helpful information about the absence of risk. Zero-event trials:

“provide relevant data by showing that event rates for both the intervention and control groups are low and relatively equal. Excluding such trial data potentially increases the risk of inflating the magnitude of the pooled treatment effect.”

J.O. Friedrich, N.K. Adhikari, J. Beyene, “Inclusion of zero total event trials in meta-analyses maintains analytic consistency and incorporates all available data,” 5 BMC Med. Res. Methodol. 2 (2007)[cited as Friedrich].  Zero event trials can be included in meta-analyses by using something called a standard “continuity correction,” which involves imputing events, or fractional events, in all cells of the 2 x 2 table. One approach, the zero is replaced with 0.5 and all other numbers are increased by 0.5. Friedrich at 7.

After examining the bias in several meta-analyses from excluding zero-event trials, Friedrich and colleagues recommended:

“We believe these trials [with zero events] should also be included if RR [relative risks] or OR [odds ratios] are the effect measures to provide a more conservative estimate of effect size(even if this change in effect size is very small for RR and OR), and to provide analytic consistency and include the same number of trials in the meta-analysis, regardless of the summary effect measure used. Inclusion of zero total event trials would enable the inclusion of all available randomized controlled data in a meta-analysis, thereby providing the most generalizable estimate of treatment effect.”

Friedrich at 5-6.

Wei addressed the problem of zero-event trials by using common imputation methods, not so different from what plaintiffs’ expert witness Dr. Ix used in the gadolinium litigation. See Meta-Meta-Analysis — The Gadolinium MDL — More Than Ix’se Dixit.  Given that plaintiffs advanced a mechanistic theory, which would explain cardiovascular thrombotic events almost immediately upon first ingestion of Celebrex, Professor Wei’s attempt to save the data inherent in zero-event trials by “continuity correction” or imputation methods seems reasonable and well within meta-analytic procedures.

 

RISK DIFFERENCE

Professor Wei did not limit himself to a single method or approach.  In addition to using imputation methods, Wei used risk difference, rather than risk ratios, as the parameter of interest.  The risk difference is simply the difference between two risks: the risk or probability of an event in one group less the risk or probability of that event in another group.  Contrary to the plaintiffs’ claims, there is nothing novel or subversive about conducting a meta-analysis with the risk difference as the parameter of interest, rather than a risk ratio.  In the context of randomized clinical trials, the risk difference is expected as a measure of absolute effect.  See generally, Michael Borenstein, L. V. Hedges, J. P. T. Higgins, and H. R. Rothstein, Introduction to Meta-Analysis (2009); Julian PT Higgins and Sally Green, eds., Cochrane Handbook for Systematic Reviews of Interventions (2008)

Like risk ratios, the risk difference yield a calculated confidence interval at any desired coefficient of confidence.  Confidence intervals for dichotomous events are often based upon approximate methods that build upon the normal approximation to the binomial distribution.  These approximate methods require assumptions of sample size that may not be met in cases involving sparse data.  With modern computers, calculating exact confidence intervals is not particularly difficult, and Professor Wei has published a methods paper in which he explains the desirability of using the risk difference with exact intervals in addressing meta-analyses of sparse data, such as was involved in the Celebrex litigation.  See L. Tian, T. Cai, M.A. Pfeffer, N. Piankov, P.Y. Cremieux, and L.J. Wei, “Exact and efficient inference procedure for meta-analysis and its application to the analysis of independent 2 x 2 tables with all available data but without artificial continuity correction,” 10 Biostatistics 275 (2009).

Plaintiffs attacked Wei’s approach as “novel” and not generally accepted.  Judge Swain appropriately dismissed this attack:

“Dr. Wei’s methodology, the validity of which Plaintiffs contest and the novelty of which Plaintiffs seek to highlight, appears to have survived the rigors of peer review at least once, and is subject to critique by virtue of its transparency. Dr. Wei’s report, supplemented by his declaration, is sufficient to meet Defendants’ burden of demonstrating that his testimony is the product of reliable principles and methods. He has explained his methods, which can be tested. Plaintiffs’ critiques of Dr. Wei’s choices regarding which trials to include in his own meta-analysis, the origins of the data he used, the date at which he undertook his meta-analysis, and at whose behest he performed his analysis all go to the weight of Dr. Wei’s testimony.”

In re Pfizer Inc. Securities Litig., 2010 WL 1047618, *7 (S.D.N.Y. 2010).  The approach taken by Wei is novel only in the sense that researchers have not previously tried to push the methodological envelope of meta-analysis to deploy the technique for rare outcomes and sparse data, with many zero-event trials.  The risk difference approach is well suited to the situation, and the use of exact confidence intervals is hardly novel or dubious.

Posted in Causation, Frye, Meta-analysis, Rule 702 | Comments Off on Meta-Meta-Analysis – Celebrex Litigation – The Claims – Part One

Expert Witness Guru

June 20th, 2012

In my casting about on the internet, I came across an interesting site dedicated to expert witness issues:  The Expert Witness Guru (EWG).

As you might expect, the Expert Witness Guru is an India-based “expert witness consulting and publishing firm, with a distinct focus on helping experts build a robust practice and providing attorneys with the right tools to locate and engage the right experts.”  The website is a blend of marketing and scholarship, which is designed to be helpful to both expert witnesses and to lawyers who depend so much upon their experts in litigation.  The EWG offers a variety of litigation support services, including preparation of expert witness profiles and deposition summaries.

The website features a blog, the Expert Witness Marketing and News Blog, which is designed with the mixed readership of both expert witnesses and lawyers in mind.  The EWG blog publishes the work of the folks at EWG, as well as invited guest posts.  Expert Witness Guru also publishes an electronic monthly magazine, Expert Witness Chronicle, with coverage of the law, practice, and marketing of expert witness testimony.  To date, two issues have been released, which are available at the EWG website.

The editorial staff of Expert Witness Chronicle includes Ashish Arun (Editor), Shweta Nawani (Co-Editor), and contributors John F. Fielder, Myles Levin, and Gil Zamora.

The second issue of the Expert Witness Chronicle announced the formation of a capable editorial board, of well-respected scholars and practictioners:

Joseph P. Sanders, the A. A. White Professor of Law at University of Houston Law Center

Edward K. Cheng, Professor of Law at Vanderbilt Law School, and

John F. Fielder, clinical and forensic psychologist, and CEO, Daubert Institute of Forensic Psychology

The first two issues feature interesting coverage of the whole range of expert witness issues, from practice issues involving retention to the wide spectrum of types of expert witness testimony (forensic, economic, engineering, scientific, etc.).  The website, blog, magazine, and the outsourced services are all worth a closer look.

 

 

Posted in Expert Witnesses | Comments Off on Expert Witness Guru

NIOSH Report Sets Up Run on September 11th Victim Compensation Fund by Non-Victims

June 16th, 2012

Congress created September 11th Victim Compensation Fund, 49 USC § 40101, to compensate victims of the terrorist attack.  Being a victim implies that the harm to be compensated was caused by the attack and its consequences.  Understandably, many of the harms were acute injuries, but what about cancer?  The latency period for most cancers are greater than 10 years, and the latency alone would suggest that persons who developed cancer within 10 years were not “victims,” but rather expected incidences of prevalent, chronic disease, or the result of much earlier exposures in the patients’ lifetimes.

Less than a year ago, the New York Times reported on a NIOSH report, which documented that there was little evidence upon which to rely, and what was available did not support conclusions of causality.  See Anemona Hartocollis, “Scant Evidence to Link 9/11 to Cancer, U.S. Report Says,” N.Y. Times (July 26, 2011).  The report appropriately noted that “[d]rawing causal inferences about exposures resulting from the Sept. 11, 2001, terrorist attacks and the observation of cancer cases in responders and survivors is especially challenging since cancer is not a rare disease.”

A few months later, the Times reported on an epidemiologic study of firefighters who were present at the World Trade Center in 2001.  Sydney Ember, “Study Suggests Higher Cancer Risk for 9/11 Firefighters,” N.Y. Times (Sept. 1, 2011).  According to the Times, the study:

“says firefighters who toiled in the wreckage of the World Trade Center in 2001 were 19 percent more likely to develop cancer than those who were not there, the strongest evidence to date of a possible link between work at ground zero and cancer. The study, published Thursday in the British medical journal The Lancet, included almost 10,000 New York City firefighters, most of whom were exposed to the caustic dust and smoke created by the fall of the twin towers. The findings indicate an “increased likelihood for the development of any type of cancer,” said Dr. David J. Prezant, the chief medical officer for the New York Fire Department, who led the study. But he said the results were far from conclusive. ‘This is not an epidemic’, he said.”

Well this is just bad reporting; the study said nothing of the sort.  The study reported a non-statistically significant standardized incidence ratio for all cancer, of either 1.10 (95% CI 0·98–1·25), with a comparison group of the generalized U.S. male population, or 1·19 (95% CI 0·96–1·47), with unexposed firefighters as a comparison group, and corrected for possible surveillance bias.  Here are the authors’ (including Dr. Prezant’s) published interpretation of the data:

“We reported a modest excess of cancer cases in the WTC-exposed cohort. We remain cautious in our interpretation of this finding because the time since 9/11 is short for cancer outcomes, and the reported excess of cancers is not limited to specific organ types. As in any observational study, we cannot rule out the possibility that effects in the exposed group might be due to unidentified confounders. Continued follow-up will be important and should include cancer screening and prevention strategies.

Rachel Zeig-Owens, Mayris Webber, Charles Hall, Theresa Schwartz, Nadia Jaber, Jessica Weakle , Thomas Rohan, Hillel Cohen, Olga Derman, Thomas Aldrich, Kerry Kelly, David  Prezant, “Early assessment of cancer outcomes in New York City firefighters after the 9/11 attacks: an observational cohort study,” 378 Lancet 898, 898 (2011) [hereafter Zeig-Owens].

The Zeig-Owens study was a cohort of New York firefighters who had worked at the WTC in the immediate aftermath of the attack.  The data neither ruled out chance nor bias and confounding as a basis for the reported risk ratios. The potentially toxic exposures at the WTC were only some of the exposures these men experienced over their careers.  Comparisons with the general population are thus not terribly revealing, but the study also compared the firefighters with other firefighters who did not work at the WTC.

For firefighters, lung cancer is typically a concern, but the Zeig-Owens study reported that the WTC firefighters had a lower than expected incidence of lung cancer:

lung cancer 0.53 (95% CI 0.18 – 1.54)

Some cancers had an elevated SIR, which is also expected given that the study looked at dozens of different outcomes.  Esophageal cancer was typical of the few that cancers that fell above 1.0; the SIR was 1.32, but the 95% confidence interval was huge, running from 0.12 to 14.53.  Understandably the authors of the WTC study did not assert any causal conclusions.

The lack of causal conclusions and evidence did not ultimately stand in the way of politics.  Last week, John Howard, the director of the National Institute of Occupational Safety and Health (NIOSH) issued his ruling that some 50 different types of cancer be added to the illnesses and injuries covered by the 9/11 Compensation Fund.  Anemona Hartocollis, “Sept. 11 Health Fund Given Clearance to Cover Cancer,” N.Y. Times (June 8, 2012).

Not only is Howard’s report not based upon appropriate scientific conclusions of causality, it is a ghastly insult to those men and women who were truly victims of the attack.  On virtually no evidence at all, Howard’s decision dilutes the fund for those truly injured.  The extent of the dilution is disturbing; the decision will not only allow the victims and the heroic rescuers to apply for compensation for cancers, but it will allow residents and passerbys to do so, as well.

The Times quoted Dr. Howard as stating that the Zeig-Owens study provided “a strong foundation for a conclusion that some cancers had been caused by exposure to the WTC debris.”  This is rubbish.  Coming from the director of a supposedly scientific agency, the statement is shocking.  As noted above, the lung cancer incidence ratio for WTC firefighters was lower than expected, compared to either non-WTC firefighters or the general male population.  The presence of “known or potential carcinogens” in the debris hardly justifies compensation unless the exposures were at sufficient intensity and duration, with appropriate latencies, to have caused the claimed cancers.  Howard’s report is shamelessly bereft of evidence to support bilking the compensation fund, and diverting compensation from true victims of the jihadist attack.

Although the Times ignored the primary data, it did acknowledge that Howard’s report, and the recommendation upon which he relied, seemed to be based upon “societal concerns that the cancer patients not be left out of the fund.”  This is interest-group politics substituting for science.

The Times quoted Dr. Alfred I. Neugut, an oncologist and professor of epidemiology at the Mailman School of Public Health at Columbia University, as stating that the decision was “primarily motivated by concern for a sympathetic population,”  and that “[t]he scientific evidence currently is certainly weak; whether future evidence bears out the wisdom of this decision will have to be seen.”

But “weak” is understatement.  The Zeig-Owens study looked at dozens of organ cancers and subtypes; it was a huge exercise in data mining, which can best be described as hypothesis generating.  Howard, however, decided to reject any semblence of evidence-based medicine:

“Requiring evidence of positive associations from studies of 9/11-exposed populations exclusively does not serve the best interests.”

So apparently positive associations are no longer required; compensation can be based upon negative associations, as was the case with WTC firefighting and lung cancer.

The Times cheered Howard’s decision in an editorial that followed quickly on the heels of the NIOSH report. “Ground Zero Cancers” (June 14, 2012).  The Times acknowledged that “[s]ome experts still believe the evidence linking the Sept. 11 attacks to cancers is weak. But we have a moral obligation to ensure that those harmed by exposure at ground zero get the medical and financial help they need.”

But “weak” is a gross overstatement, and whence comes a moral obligation to help those not harmed by exposure at ground zero?  Where is the morality of diluting the compensation fund for those who were truly victims.  Howard’s report represents not only an abdication of the evidence-based world view, but profound disrespect for those were killed and maimed in this brutal attack.

Predictably, plaintiffs’ counsel have already urged mesothelioma patients to consider the fund.  See “Lawyer Urges NYC Mesothelioma Sufferers to Explore Options after Decision Expanding 9/11 Fund,” New York, NY (PRWEB) (June 16, 2012) (“New York mesothelioma lawyer Joseph W. Belluck today said that 9/11 workers with mesothelioma should step forward to explore their eligibility for compensation in light of a federal ruling that greatly expands the scope of a $4.3 billion fund established to compensate and treat people exposed to toxic smoke, dust and fumes following the Sept. 11, 2001, terrorist attacks.”) No mesothelioma cases were reported as incident among the WTC exposed firefighters.

An internet search on ” cancer 9/11 compensation fund” turned up dozens of lawyer advertisements and websites urging cancer claims against the Fund.

Posted in Uncategorized | Comments Off on NIOSH Report Sets Up Run on September 11th Victim Compensation Fund by Non-Victims

The WABAC on the Wayback Machine – Proving Up Internet History

June 16th, 2012

Every TV-literate American of my generation knows that Mr. Peabody and his boy, Sherman, invented a time machine, the “WABAC machine,” which allowed them to travel back in time to explore, and to alter, historical events. The Rocky and Bullwinkle Show. My children know that the universe began with the creation of the internet, and that all you need to visit the past is the Wayback Machine, at http://archive.org/index.php

The content of the internet is not static; webpages come and go.  The Wayback Machine periodically archives snapshots of webpages, and makes them available for review.  The Wayback Machine has great utility for investigative journalists, historians, and, of course, lawyers.  See Healthcare Advocates, Inc. v. Harding, Earley, Follmer & Frailey, 497 F. Supp. 2d 627 (E.D. Pa. 2007) (describing the Internet Archive as “a nonprofit organization that has created an online library of digital media in an effort to preserve digital content for future reference. Its digital database is equivalent to a paper library, but is filled with digital media like websites instead of books. The library includes a collection of chronological records of various websites which Internet Archive makes available at no cost to the public via the Wayback Machine. The library’s records include more than 85 billion screenshots of web pages which are stored on a computer database in California. Internet Archive’s database provides users with the ability to study websites that may have been changed or no longer exist.”).

A few years ago, when I was practicing in Philadelphia, an out-of-town firm filed over 100 silicosis cases in the Court of Common Pleas.  The cases were generated by screenings funded and organized by the plaintiffs’ firm.  The plaintiffs’ and defense counsel tussled over the propriety of the screenings, and the plaintiffs’ firm insisted that it did not historically conduct radiographic screenings for pneumoconiosis cases.  A simple search for the plaintiffs’ firm’s website, on The Wayback Machine, told a different story.   See Schachtman, “State Regulators Impose Sanction for Unlawful Silicosis Screenings,” 17(13) Wash. Leg. Fdtn. Leg. Op. Ltr. (May 25, 2007).

Proving the past content of the plaintiffs’ firm’s website never became an issue in the Philadelphia silicosis issue, but the Wayback Machine has figured in other litigation where authentication was required.  United States v. Bansal, 663 F.3d 634 (3d Cir. 2011);  Keystone Retaining Wall Sys., Inc. v. Basalite Concrete Prods., LLC, 2011 U.S. Dist. LEXIS 145545, n.9 (D. Minn. Dec. 19, 2011) (noting that federal courts have regularly accepted evidence from the Internet Archive); St. Luke’s Cataract & Laser Inst., P.A. v. Sanderson, No. 8:06-CV-223, 2006 U.S. Dist. LEXIS 28873, at *6, 2006 WL 1320242, at *2 (M.D. Fla. May 12, 2006) (noting that a screen capture from the Internet Archive could be printed out and authenticated with an affidavit from a “representative of Internet Archive with personal knowledge of its contents, verifying that the printouts Plaintiff seeks to admit are true and accurate copies of Internet Archive’s records”).

There is also a significant secondary literature describing authentication procedures.  See WAYBACK MACHINE MEMO: Report of the Discovery Practices and Procedures Subcommittee of the Enforcement Committee (Nov. 2009) (collecting cases and urging authentication by judicial notice or stipulation); Wayback Machine Frequently Asked Questions (describing model affidavit for authentication, and alternative procedures in lieu of affidavit).

See also Kenneth N. Rashbaum, Matthew F. Knouff, and Dominique Murray, “Admissibility of Non-U.S. Electronic Evidence,” 18 Richmond J. L. & Tech. 9 (2012);  Rebecca Levy-Sachs & Jason Curtin, “Clearing Hurdles to Admission,” For The Defense 24 (Jan. 2011); James Gibson & Ketan Bhirud, “Admitting Web Pages Into Evidence,” Nevada Lawyer 15 (Oct. 2010); Deborah R. Eltgroth, “Best Evidence and the Wayback Machine: Toward a Workable Authentication Standard for Archived Internet Evidence,” 78 Fordham L. Rev. 181 (2009); Beryl A. Howell, “Proving Web History: How to Use the Internet Archive,” J. Internet Law 3 (Feb. 2006); Gregory P. Joseph, “Internet Archive (Wayback Machine) Printouts Received in Evidence on Preliminary Injunction Motion Evidently without Further Authentication — Courts Warming to Reliability“; Federal Evidence Review, “Authenticating Internet Screenshot Evidence Under FRE 901” (Dec. 19, 2011) (describing Bansal case and its review of authentication methods for the Wayback Machine archive of website pages).

 

Posted in Historian Testimony | Comments Off on The WABAC on the Wayback Machine – Proving Up Internet History

The Cherry-Picking Fallacy in Synthesizing Evidence

June 15th, 2012

What could be wrong with picking cherries?  At the end of the process you have cherries, and if you do it right, you have all ripe, and no rotten, cherries.  Your collection of ripe cherries, however, will be unrepresentative of the universe of cherries, but at least we understand how and why your cherries were selected.

Elite colleges pick the best high school students; leading law schools pick the top college students; and top law firms and federal judges cherry pick the best students of the best law schools.  Lawyers are all-too-comfortable with “cherry picking.”  Of course, the cherry-picking process here has at least some objective criteria, which can be stated in advance of the selection.

In litigation, each side is expected to “cherry pick” the favorable evidence, and ignore or flyblow the contrary evidence.  Judges are thus often complacent about selectivity in the presentation of evidence by parties and their witnesses.  In science, this kind of adversarial selectivity is a sure way to inject bias and subjectivity into claims of knowledge.  The development of the systematic review, in large measure, has been supported by the widespread recognition that studies cannot be evaluated with post hoc, subjective evaluative criteria. Cynthia D. Mulrow, Deborah J. Cook, Frank Davidoff, “Systematic Reviews: Critical Links in the Great Chain of Evidence,” 126 Ann. Intern. Med. 389 (1997)

The International Encyclopedia of Philosophy describes “cherry picking” as a fallacy, “a kind of error in reasoning.”  Cherry-picking the evidence, also known as “suppressed evidence,” is:

“[i]ntentionally failing to use information suspected of being relevant and significant is committing the fallacy of suppressed evidence. This fallacy usually occurs when the information counts against one’s own conclusion. * * * If the relevant information is not intentionally suppressed but rather inadvertently overlooked, the fallacy of suppressed evidence also is said to occur, although the fallacy’s name is misleading in this case.”

Bradley Dowden, “Suppressed Evidence,” International Encyclopedia of Philosophy (Last updated: December 31, 2010).

Cherry picking is a main rhetorical device for the litigator, and many judges simply do not understand what is so wrong with each side’s selection of the studies that it wishes to emphasize.  Whatever the acceptability of lawyers’ cherry picking in the presentation of evidence, it is antithetical to scientific methodology.  “Cherry picking (fallacy),” Wikipedia (describing cherry picking as the pointing to data that appears to confirm one’s opinion, while ignoring contradictory data)[last visited on June 14, 2012]

Given the pejorative connotations of “cherry picking,” no one should be very surprised that lawyers and judges couch their Rule 702 arguments and opinions in terms of whether expert witnesses engaged in this fruitful behavior.  Although I had heard plaintiffs’ and defendants’ counsel use the phrase, I only recently came across it in a judicial opinion.  Since the phrase nicely describes a fallacious form of reasoning, I thought it would be helpful to collect pertinent cases that describe the fallaciousness of fruit-pickin’ expert witness testimony.

United States Court of Appeals

Barber v. United Airlines, Inc., 17 Fed.Appx. 433, 437 (7th Cir. 2001) (affirming exclusion of “cherry-picking” expert witness who failed to explain why he ignored certain data while accepting others)

District Courts

Dwyer v. Sec’y of Health & Human Servs., No. 03-1202V, 2010 WL 892250 (Fed. Cl. Spec. Mstr. Mar. 12, 2010)(recommending rejection of thimerosal autism claim)(“In general, respondent’s experts provided more responsive answers to such questions.  Respondent’s experts were generally more careful and nuanced in their expert reports and testimony. In contrast, petitioners’ experts were more likely to offer opinions that exceeded their areas of expertise, to “cherry-pick” data from articles that were otherwise unsupportive of their position, or to draw conclusions unsupported by the data cited… .”)

In re Bausch & Lomb, Inc., 2009 WL 2750462 at 13 (D. S.C. 2009)( “Dr. Cohen did not address [four contradictory] studies in her expert reports or affidavit, and did not include them on her literature reviewed list [. . .] This failure to address this contrary data renders plaintiffs’ theory inherently unreliable.”)

Rimbert v. Eli Lilly & Co., No. 06-0874, 2009 WL 2208570, *19 (D.N.M. July 21, 2009) )(“Even more damaging . . . is her failure to grapple with any of the myriad epidemiological studies that refute her conclusion.”), aff’d, 647 F.3d 1247 (10th Cir. 2011) (affirming exclusion but remanding to permit plaintiff to find a new expert witness)

In re Bextra & Celebrex Prod. Liab. Litig., 524 F. Supp.2d 1166, 1176, 1179, 1181, 1184 (N.D. Cal. 2007) (criticizing plaintiffs’ expert witnesses for “cherry-picking studies”); id. at 1181 (“these experts ignore the great weight of the observational studies that contradict their conclusion and rely on the handful that appear to support their litigation-created opinion.”)

LeClerq v. Lockformer Co., No. 00 C 7164, 2005 U.S. Dist. LEXIS 7602, at *15 (N.D. Ill. Apr. 28, 2005) (holding that expert witness’s “cherry-pick[ing] the facts he considered to render his opinion, and such selective use of facts fail[s] to satisfy the scientific method and Daubert.”)(internal citations and quotations omitted)

Holden Metal & Aluminum Works v. Wismarq Corp., No. 00 C 0191, 2003 WL 1797844, at *2 (N.D. Ill. Apr. 2, 2003).

State Courts

Betz v. Pneumo Abex LLC, 2012 WL 1860853, *16 (May 23, 2012 Pa. S. Ct.)(“According to Appellants, moreover, the pathologist’s self-admitted selectivity in his approach to the literature is decidedly inconsistent with the scientific method. Accord Brief for Amici Scientists at 17 n.2 (“‘Cherry picking’ the literature is also a departure from ‘accepted procedure’.”)).

George v. Vermont League of Cities and Towns, 2010 Vt. 1, 993 A.2d 367, 398 (Vt. 2010)(expressing concern about how and why plaintiff’s expert witnesses selected some studies to include in their “weight of evidence” methodology.  Without an adequate explanation of selection and weighting criteria, the choices seemed arbitrary)

Scaife v. AstraZeneca LP, 2009 WL 1610575 at 8 (Del. Super. 2009) (“Simply stated, the expert cannot accept some but reject other data from the medical literature without explaining the bases for her acceptance or rejection.”)

In re Bextra & Celebrex, 2008 N.Y. Misc. LEXIS 720, *20, 239 N.Y.L.J. 27 (2008) (holding that New York’s Frye rule requires proponent to show that its expert witness had “look[ed] at the totality of the evidence and [did] not ignore contrary data.”); see also id. at *36 (“Moreover, out of 32 studies (29 published) cited by defendants, plaintiffs chose only 8 to plead their case.  This smacks of ‘cherry-picking,’ skewing their analysis by only looking at the helpful studies. Such practice contradicts the accepted method for an expert’s analysis of epidemiological data.”)

Bowen v. E.I. DuPont de Nemours & Co., 906 A.2d 787, 797 (Del. 2006) (noting that expert witnesses cannot ignore studies contrary to their opinions)

Selig v. Pfizer, Inc., 185 Misc. 2d 600, 607, 713 N.Y.S.2d 898 (Sup. Ct. N.Y. Cty. 2000) (holding that expert witness failed to satisfy Frye test’s requirement of following an accepted methodology when he ignored studies contrary to his opinion), aff’d, 290 A.D.2d 319, 735 N.Y.S.2d 549 (1st Dep’t 2002)

******************

Most but not all the caselaw uniformly recognizes the fallacy for an expert witness to engage in ad hoc selectivity in addressing studies upon which to rely.  In the following two cases, the cherry-picking was identified, but acquiesced in by judges.

McClellan v. I-Flow Corp., 710 F. Supp. 2d 1092, 1114 (D. Ore. 2010)(discussing cherry picking but rejecting “document by document” review)(“Finally, defendants contend that plaintiffs’ experts employ unreliable methodologies by ‘cherry-picking’ facts from certain studies and asserting reliance on the ‘totality’ or ‘global gestalt of medical evidence’. Defendants argue that in  doing so, plaintiffs’ experts fail to ‘painstakingly’ link each piece of data to their conclusions or explain how the evidence supports their opinions.”)

United States v. Paracha, 2006 WL 12768 (S.D. N.Y. Jan. 3, 2006)(rejecting challenge to terrorism expert on grounds that he cherry picked evidence in conspiracy prosecution involving al Queda)

King v. Burlington No. Santa Fe Ry, ___N.W.2d___, 277 Neb. Reports 203, 234 (2009)(noting that the law does “not preclude a trial court from considering as part of its reliability inquiry whether an expert has cherry-picked a couple of supporting studies from an overwhelming contrary body of literature,” but ignoring the force of the fallacious expert witness testimony by noting that the questionable expert witness (Frank) had some studies that showed associations between exposure to diesel exhaust or benzene and multiple myeloma).

Posted in Causation, Rule 702 | Comments Off on The Cherry-Picking Fallacy in Synthesizing Evidence

Another Confounder in Lung Cancer Occupational Epidemiology — Diesel Engine Fumes

June 13th, 2012

Researchers obviously need to be aware of, and control for, potential and known confounders.  In the context of investigating the etiologies of lung cancer, there is a long list of potential confounding exposures, often ignored in peer-reviewed papers, which focus on one particular outcome of interest.  Just last week, I wrote to emphasize the need to account for potential and known confounding agents, and how this need was particularly strong in studies of weak alleged carcinogens such as crystalline silica.  See Sorting Out Confounded Research – Required by Rule 702.  Yesterday, the World Health Organization (WHO) added another “known” confounder for lung cancer epidemiology:  diesel fume.

According to the International Agency for Research on Cancer (IARC), a division of the WHO, a working group of international experts voted to reclassify diesel engine exhaust as a “Group I” carcinogen.  IARC: Diesel engines exhaust carcinogenic (2012).  This classification means, in IARC parlance, that ” there is sufficient evidence of carcinogenicity in humans. Exceptionally, an agent may be placed in this category when evidence of carcinogenicity in humans is less than sufficient but there is sufficient evidence of carcinogenicity in experimental animals and strong evidence in exposed humans that the agent acts through a relevant mechanism of carcinogenicity.”  The Group was headed up by Dr. Christopher Portier, who is the director of the National Center for Environmental Health and the Agency for Toxic Substances and Disease Registry at the Centers for Disease Control and Prevention.  Id.

The reclassification removes diesel exhaust from its previous categorization as a Group 2A carcinogen, which is interpreted “as probably carcinogenic to humans.”  Diesel exhaust has been on a high-priority list for re-evaluation since 1998, as result of epidemiologic research from many countries.  The Working Group specifically found that there was sufficient evidence to conclude that diesel exhaust is a cause of lung cancer in humans, and limited evidence to support an association with bladder cancer.  The Group rejected any change in classification of gasoline engine exhaust from its current IARC rating as “possibly carcinogenic to humans. (Group 2B).”

Unlike other IARC Working Group decisions (such as crystalline silica), which were weakened by close votes and significant dissents, the diesel Group’s conclusion was unanimous.  The diesel Group appeared to be impressed by two recent studies of lung cancer in underground miners, released in March 2012.  One study was in a large cohort, conducted by NIOSH, and the other was a nested case-control study, conducted by the National Cancer Institute (NCI).  See Debra T. Silverman, Claudine M. Samanic, Jay H. Lubin, Aaron E. Blair, Patricia A. Stewart , Roel Vermeulen, Joseph B. Coble, Nathaniel Rothman, Patricia L. Schleiff , William D. Travis, Regina G. Ziegler, Sholom Wacholder, Michael D. Attfield, “The Diesel Exhaust in Miners Study: A Nested Case-Control Study of Lung Cancer and Diesel Exhaust,” J. Nat’l Cancer Instit. (2012)(in press and open access); and Michael D. Attfield, Patricia L. Schleiff, Jay H. Lubin, Aaron Blair, Patricia A. Stewart, Roel Vermeulen, Joseph B. Coble, and Debra T. Silverman, “The Diesel Exhaust in Miners Study: A Cohort Mortality Study With Emphasis on Lung Cancer,” J. Nat’l Cancer Instit. (2012)(in press).

According to a story in the New York Times, the IARC Working Group described diesel engine exhaust as “more carcinogenic than secondhand cigarette smoke.”  Donald McNeil, “W.H.O. Declares Diesel Fumes Cause Lung Cancer,” N.Y. Times (June 12, 2012).  The Times also quoted Dr. Debra Silverman, NCI chief of environmental epidemiology, at length.  Dr. Silverman, who was the lead author of the nested case-control study cited by the IARC Press Release, noted that her large study showed that long-term heavy exposure to diesel fumes increased lung cancer risk seven fold. Dr. Silverman described this risk as much greater than that thought to be created by passive smoking, but much smaller than smoking two packs of cigarettes a day.  She stated that “totally” supported the IARC reclassification, and that she believed that governmental agencies would use the IARC analysis as the basis for changing the regulatory classification of diesel exhaust.

Silverman’s nested case-control study appears to have been based upon careful diesel exhaust exposure information, as well as smoking histories.  The study also searched and analyzed for other potential confounders, which might be expected to be involved in underground mining:

“Other potential confounders [ie, duration of cigar smoking; frequency of pipe smoking; environmental tobacco smoke; family history of lung cancer in a first-degree relative; education; body mass index based on usual adult weight and height; leisure time physical activity; diet; estimated cumulative exposure to radon, asbestos, silica, polycyclic aromatic hydrocarbons (PAHs) from non-diesel sources, and respirable dust in the study facility based on air measurement and other data (14)] were evaluated but not included in the final models because they had little or no impact on odds ratios (ie, inclusion of these factors in the final models changed point estimates for diesel exposure by ≤ 10%).”

Silverman, et al., at 4.  The absence of an association between lung cancer and silica exposure is noteworthy in a such a large study of underground miners.

Posted in Causation, Risk and Risk Factor, Rule 702, Scientific Evidence, Silica | Comments Off on Another Confounder in Lung Cancer Occupational Epidemiology — Diesel Engine Fumes

Sorting Out Confounded Research – Required by Rule 702

June 10th, 2012

CONFOUNDING

Back in 2000, several law professors wrote an essay, in which they detailed some of the problems faced in expert witness gatekeeping.  They noted that judges easily grasped the problem of generalizing from animal evidence to human experience, and thus they simplistically emphasized human (epidemiologic) data.  But in their emphasis, the judges missed problems of internal validity, such as confounding, in epidemiologic studies:

“Why do courts have such a preference for human epidemiological studies over animal experiments? Probably because the problem of external validity (generalizability) is one of the most obvious aspects of research methodology, and therefore one that non-scientists (including judges) are able to discern with ease – and then give excessive weight to (because whether something generalizes or not is an empirical question; sometimes things do and other times they do not). But even very serious problems of internal validity are harder for the untrained to see and understand, so judges are slower to exclude inevitably confounded epidemiological studies (and give insufficient weight to that problem). Sophisticated students of empirical research see the varied weaknesses, want to see the varied data, and draw more nuanced conclusions.”

David Faigman, David Kaye, Michael Saks, Joseph Sanders, “How Good is Good Enough?  Expert Evidence Under Daubert and Kumho,” 50 Case Western Reserve L. Rev. 645, 661 n.55 (2000).  I am not sure that the problems are dependent in the fashion suggested by the authors, but their assessment that judges may be slow and frequently lack the ability to draw nuanced conclusions seems fair enough. Judges continue to miss important validity issues, perhaps because the adversarial process levels all studies to debating points in litigation.  See, e.g., In re Welding Fume Prods. Liab. Litig., 2006 WL 4507859, *33 (N.D.Ohio 2006)(reducing all studies to one level, and treating all criticisms as though they rendered all studies invalid).

[This discussion of confounding has been updated; see here and there.]

 

Posted in Uncategorized | Comments Off on Sorting Out Confounded Research – Required by Rule 702

Meta-Meta-Analysis — The Gadolinium MDL — More Than Ix’se Dixit

June 8th, 2012

There is an tendency, for better or worse, for legal bloggers to be partisan cheerleaders over litigation outcomes.  I admit that most often I am dismayed by judicial failures or refusals to exclude dubious plaintiffs’ expert witnesses’ opinion testimony, and I have been known to criticize such decisions.  Indeed, I wouldn’t mind seeing courts exclude dubious defendants’ expert witnesses.  I have written approvingly about cases in which judges have courageously engaged with difficult scientific issues, seen through the smoke screen, and properly assessed the validity of the opinions expressed.  The Gadolinium MDL (No. 1909) Daubert motions and decision offer a fascinating case study of a challenge to an expert witness’s meta-analysis, an effective defense of the meta-analysis, and a judicial decision to admit the testimony, based upon the meta-analysis.  In re Gadolinium-Based Contrast Agents Prods. Liab. Litig., 2010 WL 1796334 (N.D. Ohio May 4, 2010) [hereafter Gadolinium], reconsideration denied, 2010 WL 5173568 (June 18, 2010).

Plaintiffs proffered general causation opinions (between gadolinium contrast media and Nephrogenic Systemic Fibrosis (“NSF”), by a nephrologist, Joachim H. Ix, M.D., with training in epidemiology.  Dr. Ix’s opinions were based in large part upon a meta-analysis he conducted on data in published observational studies.  Judge Dan Aaron Polster, the MDL judge, itemized the defendant’s challenges to Dr. Ix’s proposed testimony:

“The previously-used procedures GEHC takes issue with are:

(1) the failure to consult with experts about which studies to include;

(2) the failure to independently verify which studies to select for the meta-analysis;

(3) using retrospective and non-randomized studies;

(4) relying on studies with wide confidence intervals; and

(5) using a “more likely than not” standard for causation that would not pass scientific scrutiny.”

Gadolinium at *23.  Judge Polster confidently dispatched these challenges.  Dr. Ix, as a nephrologist, had subject-matter expertise with which to develop inclusionary and exclusionary criteria on his own.  The defendant never articulated what, if any, studies were inappropriately included or excluded.  The complaint that Dr. Ix had used retrospective and non-randomized studies also rang hollow in the absence of any showing that there were randomized clinical trials with pertinent data at hand.  Once a serious concern of nephrotoxicity arose, clinical trials were unethical, and the defendant never explained why observational studies were somehow inappropriate for inclusion in a meta-analysis.

Relying upon studies with wide confidence intervals can be problematic, but that is one of the reasons to conduct a meta-analysis, assuming the model assumptions for the meta-analysis can be verified.  The plaintiffs effectively relied upon a published meta-analysis, which pre-dated their expert witness’s litigation effort, in which the authors used less conservative inclusionary criteria, and reported a statistically significant summary estimate of risk, with an even wider confidence interval.  R. Agarwal, et al., ” Gadolinium-based contrast agents and nephrogenic systemic fibrosis: a systematic review and meta-analysis,” 24 Nephrol. Dialysis & Transplantation 856 (2009).  As the plaintiffs noted in their opposition to the challenge to Dr. Ix:

“Furthermore, while GEHC criticizes Dr. Ix’s CI from his meta-analysis as being “wide” at (5.18864 and 25.326) it fails to share with the court that the peer-reviewed Agarwal meta-analysis, reported a wider CI of (10.27–69.44)… .”

Plaintiff’s Opposition to GE Healthcare’s Motion to Exclude the Opinion Testimony of Joachim Ix at 28 (Mar. 12, 2010)[hereafter Opposition].

Wider confidence intervals certainly suggest greater levels of random error, but Dr. Ix’s intervals suggested statistical significance, and he had carefully considered statistical heterogeneity.  Opposition at 19. (Heterogeneity was never advanced by the defense as an attack on Dr. Ix’s meta-analysis).  Remarkably, the defendant never advanced a sensitivity analysis to suggest or to show that reasonable changes to the evidentiary dataset could result in loss of statistical significance, as might be expected from the large intervals.  Rather, the defendant relied upon the fact that Dr. Ix had published other meta-analyses in which the confidence interval was much narrower, and then claimed that he had “required” these narrower confidence intervals for his professional, published research.  Memorandum of Law of GE Healthcare’s Motion to Exclude Certain Testimony of Plaintiffs’ Generic Expert, Joachim H. Ix, MD, MAS, In re Gadolinium MDL No. 1909, Case: 1:08-gd-50000-DAP  Doc #: 668   (Filed Feb. 12, 2010)[hereafter Challenge].  There never was, however, a showing that narrower intervals were required for publication, and the existence of the published Agarwal meta-analysis contradicted the suggestion.

Interestingly, the defense did not call attention to Dr. Ix’s providing an incorrect definition of the confidence interval!  Here is how Dr. Ix described the confidence interval, in language quoted by plaintiffs in their Opposition:

“The horizontal lines display the “95% confidence interval” around this estimate. This 95% confidence interval reflects the range of odds ratios that would be observed 95 times if the study was repeated 100 times, thus the narrower these confidence intervals, the more precise the estimate.”

Opposition at 20.  The confidence interval does not provide a probability distribution of the parameter of interest; rather the distribution of confidence intervals has a probability of covering the hypothesized “true value” of the parameter.

Finally, the defendant never showed any basis for suggesting that a scientific opinion on causation requires something more than a “more likely than not” basis.

Judge Polster also addressed some more serious challenges:

“Defendants contend that Dr. Ix’s testimony should also be excluded because the methodology he utilized for his generic expert report, along with varying from his normal practice, was unreliable. Specifically, Defendants assert that:

(1) Dr. Ix could not identify a source he relied upon to conduct his meta-analysis;

(2) Dr. Ix imputed data into the study;

(3) Dr. Ix failed to consider studies not reporting an association between GBCAs and NSF; and

(4) Dr. Ix ignored confounding factors.”

Gadolinium at *24

IMPUTATION

The first point, above – the alleged failure to identify a source for conducting the meta-analysis – rings fairly hollow, and Judge Polster easily deflected it.  The second point raised a more interesting challenge.  In the words of defense counsel:

“However, in arriving at this estimate, Dr. Ix imputed, i.e., added, data into four of the five studies.  (See Sept. 22 Ix Dep. Tr. (Ex. 20), at 149:10-151:4.)  Specifically, Dr. Ix added a single case of NSF without antecedent GBCA exposure to the patient data in the underlying studies.

* * *

During his deposition, Dr. Ix could not provide any authority for his decision to impute the additional data into his litigation meta-analysis.  (See Sept. 22 Ix Dep. Tr. (Ex. 20), at 149:10-151:4.)  When pressed for any authority supporting his decision, Dr. Ix quipped that ‘this may be a good question to ask a Ph.D level biostatistician about whether there are methods to [calculate an odds ratio] without imputing a case [of NSF without antecedent GBCA exposure]’.”

Challenge at 12-13.

The deposition reference suggests that the examiner had scored a debating point by catching Dr. Ix unprepared, but by the time the parties briefed the challenge, the plaintiffs had the issue well in hand, citing A. W. F. Edwards, “The Measure of Association in a 2 × 2 Table,” 126 J. Royal Stat. Soc. Series A 109 (1963); R.L. Plackett, “The Continuity Correction in 2 x 2 Tables,” 51 Biometrika 327 (1964).  Opposition at 36 (describing the process of imputation in the event of zero counts in the cells of a 2 x 2 table for odds ratios).  There are qualms to be stated about imputation, but the defense failed to make them.  As a result, the challenge overall lost momentum and credibility.  As the trial court stated the matter:

“Next, there is no dispute that Dr. Ix imputed data into his meta-analysis. However, as Defendants acknowledge, there are valid scientific reasons to impute data into a study. Here, Dr. Ix had a valid basis for imputing data. As explained by Plaintiffs, Dr. Ix’s imputed data is an acceptable technique for avoiding the calculation of an infinite odds ratio that does not accurately measure association.7 Moreover, Dr. Ix chose the most conservative of the widely accepted approaches for imputing data.8 Therefore, Dr. Ix’s decision to impute data does not call into question the reliability of his meta-analysis.”

Gadolinium at *24.

FAILURE TO CONSIDER NULL STUDIES

The defense’s challenged including a claim that Dr. Ix had arbitrarily excluded studies in which there was no reported incidence of NSF. The defense brief unfortunately does not describe the studies excluded, and what, if any, effect their inclusion in the meta-analysis would have had.  This was, after all, the crucial issue. The abstract nature of the defense claim left the matter ripe for misrepresentation by the plaintiffs:

“GEHC continues to misunderstand the role of a meta-analysis and the need for studies that included patients both that did or did not receive GBCAs and reported on the incidence of NSF, despite Dr. Ix’s clear elucidation during his deposition. (Ix Depo. TR [Exh.1] at 97-98).  Meta-analyses such as performed by Dr. Ix and Dr. Agarwal search for whether or not there is a statistically valid association between exposure and disease event. In order to ascertain the relationship between the exposure and event one must have an event to evaluate. In other words, if you have a study in which the exposed group consists of 10,000 people that are exposed to GBCAs and none develop NSF, compared to a non-exposed group of 10,000 who were not exposed to GBCAs and did not develop NSF, the study provides no information about the association between GBCAs and NSF or the relative risk of developing NSF.”

Challenge at 37 – 38 (emphasis in original).  What is fascinating about this particular challenge, and the plaintiffs’ response, is the methodological hypocrisy exhibited.  In essence, the plaintiffs argued that imputation was appropriate in a case-control study, in which one cell contained a zero, but they would ignore a great deal of data in a cohort study with data.  To be sure, case-control studies are more efficient than cohort studies for identifying and assessing risk ratios for rare outcomes.  Nevertheless, the plaintiffs could easily have been hoisted with their own hypothetical petard.  No one in 10,000 gadolinium-exposed patients developed NSF; and no one in a control group did either.  The hypothetical study suggests that the rate of NSF is low and not different in the exposed and in the unexposed patients.  The risk ratio could be obtained by imputing an integer for the cells containing zero, and a confidence interval calculated.  The risk ratio, of course, would be 1.0.

Unfortunately, the defense did not make this argument; nor did it explore where the meta-analysis might have come out had a more even-handed methodology been taken by Dr. Ix.  The gap allowed the trial court to brush the challenge aside:

“The failure to consider studies not reporting an association between GBCAs and NSF also does not render Dr. Ix’s meta-analysis unreliable. The purpose of Dr. Ix’s meta-analysis was to study the strength of the association between an exposure (receiving GBCA) and an outcome (development of NSF). In order to properly do this, Dr. Ix necessarily needed to examine studies where the exposed group developed NSF.”

Gadolinium at *24.  Judge Polster, with no help from the defense brief, missed the irony of Dr. Ix’s willingness to impute data in the case-control 2 x 2 contingency tables, but not in the relative risk tables.

CONFOUNDING

Defendants complained that Dr. Ix had ignored the possibility that confounding factors had contributed to the development of NSF.  Challenge at 13.  Defendants went so far as to charge Dr. Ix with misleading the court by failing to consider other possible causative exposures or conditions.  Id.

Defendants never identified the existence, source, and likely magnitude of confounding factors.  As a result, the plaintiffs’ argument, based in the Reference Manual, that confounding was an unlikely explanation for a very large risk ratio was enthusiastically embraced by the trial court, virtually verbatim from the plaintiffs’ Opposition (at 14):

“Finally, the Court rejects Defendants’ argument that Dr. Ix failed to consider confounding factors. Plaintiffs argued and Defendants did not dispute that, applying the Bradford Hill criteria, Dr. Ix calculated a pooled odds ratio of 11.46 for the five studies examined, which is higher than the 10 to 1 odds ratio of smoking and lung cancer that the Reference Manual on Scientific Evidence deemed to be “so high that it is extremely difficult to imagine any bias or confounding factor that may account for it.” Id. at 376.  Thus, from Dr. Ix’s perspective, the odds ratio was so high that a confounding factor was improbable. Additionally, in his deposition, Dr. Ix acknowledged that the cofactors that have been suggested are difficult to confirm and therefore he did not try to specifically quantify them. (Doc # : 772-20, at 27.) This acknowledgement of cofactors is essentially equivalent to the Agarwal article’s representation that “[t]here may have been unmeasured variables in the studies confounding the relationship between GBCAs and NSF,” cited by Defendants as a representative model for properly considering confounding factors. (See Doc # : 772, at 4-5.)”

Gadolinium at *24.

The real problem is that the defendant’s challenge pointed only to possible, unidentified causal agents.  The smoking/lung cancer analogy, provided by the Reference Manual, was inapposite.  Smoking is indeed a large risk factor for lung cancer, with relative risks over 20.  Although there are other human lung carcinogens, none is consistently in the same order of magnitude (not even asbestos), and as a result, confounding can generally be excluded as an explanation for the large risk ratios seen in smoking studies.  It would be easy to imagine that there are confounders for NSF, especially given that it is relatively recently been identified, and that they might be of the same or greater magnitude as that suggested for the gadolinium contrast media.  The defense, however, failed to identify confounders that actually threatened the validity of any of the individual studies, or of the meta-analysis.

CONCLUSION

The defense hinted at the general unreliability of meta-analysis, with references to References Manual on Scientific Evidence at 381 (2d ed. 2000)(noting problems with meta-analysis), and other, relatively dated papers.  See, e.g., John Bailar, “Assessing Assessments,” 277 Science 529 (1997)(arguing that “problems have been so frequent and so deep, and overstatements of the strength of conclusions so extreme, that one might well conclude there is something seriously and fundamentally wrong with [meta-analysis].”).  The Reference Manual language carried over into the third edition, is out of date, and represents a failing of the new edition.  See “The Treatment of Meta-Analysis in the Third Edition of the Reference Manual on Scientific Evidence” (Nov. 14, 2011).

The plaintiffs came forward with some descriptive statistics of the prevalence of meta-analysis in contemporary biomedical literature.  The defendants gave mostly argument; there is a dearth of citation to defense expert witnesses, affidavits, consensus papers on meta-analysis, textbooks, papers by leading authors, and the like.  The defense challenge suffered from being diffuse and unfocused; it lost persuasiveness by including weak, collateral issues such as claiming that Dr. Ix was opining “only” on a “more likely than not” basis, and that he had not consulted with other experts, and that he had failed to use randomized trial data.  The defense was quick to attack perceived deficiencies, but it did not illustrate how or why the alleged deficiencies threatened the validity of Dr. Ix’s meta-analysis.  Indeed, even when the defense made strong points, such as the exclusion of zero-event cohort studies, it failed to document that such studies existed, and that their inclusion might have made a difference.

 

Posted in Causation, Meta-analysis, Reference Manual on Scientific Evidence, Rule 702, statistical evidence | Comments Off on Meta-Meta-Analysis — The Gadolinium MDL — More Than Ix’se Dixit

« Older Entries
Newer Entries »

The opinions, statements, and asseverations expressed on Tortini are my own, or those of invited guests, and these writings do not necessarily represent the views of clients, friends, or family, even when supported by good and sufficient reason.