TORTINI

For your delectation and delight, desultory dicta on the law of delicts.

Archives »

Silica Science – Junk Science is Not Limited to The Courts

December 12th, 2011

“Clowns to the left of me; Jokers to the right; here I am, stuck in the middle with you.”


David Michaels, head of OSHA, back in October, was testifying at a House congressional oversight hearing, “Workplace Safety: Ensuring a Responsible Regulatory Environment.” The Congressmen were inquiring into OSHA’s enforcement and regulatory initiatives on several fronts, including silica exposures.

This is the same David Michaels who used to be a hired expert witness for plaintiffs in toxic tort cases. See “David Michaels’ Public Relations Problem,” (Dec. 2, 2011).

Not surprisingly, when the questioning turned to silica, Michaels played the cancer card:  crystalline silica is a “known” human carcinogen.

Republican congressman Larry Bucshon (R-IN), a surgeon when he is not holding forth in Congress, found the talk of cancer to be provocative.  Buchson scolded Michaels:

“I don’t like it when people use buzz words that try to get people’s attention, and cancer is one of those.”

* * * * *

“…I’m a thoracic surgeon, so I want to focus a little bit on what you said earlier as it relates to silica dust. I’m curious about your comment about silica-dust related lung cancer, because I’ve been a thoracic surgeon for 15 years and I’ve done a lot of lung cancer surgery, and I haven’t seen one patient that’s got it from silica dust.”

A fascinating exchange for several reasons.

First, we could expect Michaels to play the cancer card, just as he has in his role as plaintiffs’ expert witness.  As we will see, his cancer evidence is not far fetched, although it is also not particularly convincing.

Second, the junk science from Congressman Buchson is distressing.  As a physician, he should know better that his experience in surgery has no relevance at all to the question whether crystalline silica can cause lung cancer.

Back in 1996, a working group of the World Health Organization’s International Agency for Research on Cancer (IARC) voted to reclassify crystalline silica, the most ubiquitous mineral on the face of Planet Earth, a known human carcinogen.  Michaels recited this “evidence,” but he failed to mention that the evidence was conflicting, as were the votes of the working group. The response of the scientific community to the IARC pronouncement was highly critical.  See Patrick A. Hessel, John F. Gamble, J. Bernard L. Gee, Graham Gibbs, Francis H.Y. Green, W. Keith C. Morgan, and Brooke T. Mossman, “Silica, Silicosis, and Lung Cancer: A Response to a Recent Working Group Report,” 42 J. Occup. Envt’l Med. 704 (2000).

The vote of the working group was very close; indeed, the swing of a single vote would have changed the outcome. One of the working group members later wrote:

“Some equally expert panel of scientists presented with the same information on another occasion could of course have reached a different verdict. The evidence was conflicting and difficult to assess and such judgments are essentially subjective.”

Corbett McDonald & Nicola Cherry, “Crystalline Silica and Lung Cancer:  The Problem of Conflicting Evidence,” 8 Indoor Built Environment 121, 121 (1999).  Remarkably, this panel member explained his decision to vote for reclassification as follows:

“The basic problem was that the evidence for carcinogenicity was conflicting – generally absent in situations of high and widespread exposure and strong only in a few rather special occupations.  The advice by the IARC to consider hazard rather than risk did much to resolve the difficulty.”

Id. at 125.  I suspect that the evidence for a difference in meaning between “hazard” and “risk” is even more tenuous and conflicting than the evidence in favor of carcinogenicity.

IARC classifications, however, take on a life of their own.  They are an invitation to stop thinking, and to stop analyzing the evidence.  Federal bureaucrats and staff scientists love them for exactly this reason:  they can hide behind the authority of the WHO without having to work on reviewing the evidence, or updating their judgment when new studies come out.

It should not be surprising, therefore, that the National Institutes of Health’s National Toxicology Program (NTP), working off the WHO decision, recognized crystalline silica as a human carcinogen. Other groups followed in lock step.  Other agencies and medical groups followed.

What you will not hear from Michaels or his followers is that when the National Institute for Occupational Safety and Health conducted the largest mortality study on the issue, it found a decreased lung cancer risk among men who actually had sufficient silica exposure to develop silicosis. See Geoffrey Calvert, et al., “Occupational silica exposure and risk of various diseases:  an analysis using death certificates from 27 states of the United States,” 60 Occup. Envt’l Med. 122 (2003).  Cf. “Congressman tells OSHA chief not to use “buzz” words like cancer.” (Oct. 10, 2011).

To give the devil his due, at least Michaels had “some” evidence to support his pronouncement, even if the evidence was incomplete and contradicted by other important evidence.  Congressman Bucshon’s recitation of his experience as a surgeon was completely off the mark.  His staffers obviously failed him in their research, and Bucshon’s reliance upon his own anecdotal experience was quite inappropriate to rebut the dubious judgment of the OSHA Administrator.

Some people might describe the exchange between Bucshon and Michaels as resembling two monkeys playing chess.  I think of it as exemplifying the scientific illiteracy in all three branches of our government.

David Michaels’ Public Relations Problem

Posted in Conflicts of Interest, Scientific Evidence | No Comments »

Scientific American(s) and the other 99%

December 7th, 2011

If you have an interest in the history of science, especially as it plays out in the so-called state-of-the-art defense in products liability litigation, you may find the following offer helpful.  For the remainder of the month, Scientific American, which is now published by Nature, is making its archived issues, 1845-1909, available free of charge.

There is more fascinating than to read what people were thinking, saying, and writing, at times past.  Most of what we think we know about the past is filtered by historians rather than being obtained by accessing primary sources.  The Scientific American archive is a useful corrective measure, especially in the contentious area of health-effects litigation.

Here are some of the interesting historical insights.  In 1871, 140 years ago, Scientific American ran an article on the ill-health effects of smoking.  “To smoke or not to smoke,” Scientific American 375 (Dec. 9, 1871).  Here are some highlights:

“M. Beau notices eight cases of angina pectoris caused by the use of tobacco.

Professor Lizars records several cases of cancer of the tongue and lips caused by the use of the pipe. The writer has known one such ill stance, and never wishes to see another example of such terrible suffering resulting from a worse than useless habit.”

These pronouncements might not pass muster under today’s evidence-based medicine, but they were astute observations in need of testing, in 1871.

Not all the medical observations and claims were equally prescient.  Our forebears were not immune from the idiocies and enthusiasms of medical quackery.  Cancer remedies seemed to be a particular focus of much unenlightened attention:

“Col. Ussery, of the parish of De Soto, informs the Editor of the Caddo Gazette that he fully tested a remedy for this troublesome disease, recommended to him by a Spanish woman, a native of the country. The remedy is this:  Take an egg and break it, then pour out the white, retaining the yolk in the shell, put in salt and mix with the yolk as long as it will receive it, stir them together until the salve is formed, put a portion of this on a piece of sticking plaster and apply it to the cancer about twice a day. He has made the experiment twice in his own family with complete success.”

“Remedy for Cancer,” Scientific American 298 (June 12, 1847).

Or this forerunner of the clinical trial:

“The Tuscaloosa Observer says it has seen it stated, more than once, that the common cranberry was efficacious in the cure of cancer, but have never, until very recently, been an eye-witness to the fact. Mr. Middleton Belk, residing within four or five miles of this city, who was afflicted with a cancer on the nose for the last eight years, was induced to try cranberries applied as a poultice; and to his great joy and satisfaction, has experienced a perfect and radical cure. We mention this fact at the instanee of Mr. Belk, who is desirous that others suffering under the same affliction, may avail themselves of this simple, but valuable remedy.”

“Cranberries a Cure for Cancer,” 3 Scientific American 408 (Sept. 9, 1848).  Another article, three years later, touted mineral naptha as a cancer cure.  “Mineral Naptha,” 6 Scientific American 243 (April 19, 1851).

The pages of Scientific American document the rise of asbestos use and the growing awareness of asbestos’ great utility to help control and prevent fire and burns.  For instance, in 1876, the magazine described the utility of asbestos in roofing materials and in pipecovering.  “The Industrial Uses of Asbestos,” Scientific American 258 (April 22, 1876).

A few years later, an article described the widespread use of asbestos in industrial applications, both in Europe and in the United States:

“For some time past Toope’s covering for steam surfaces has been in use in England, giving great satisfaction and receiving the indorsement of many prominent English engineers.  The business of manufacturing and selling it is conducted there by a limited company located in London.
In this country Mr. Charles Toope, manufacturing agent, having an office and works at 353 East 78th street, New York City, is making and introducing the covering.  The covering is readily applied, requires no previous preparation, and when in place is permanent, being incapable of injury by jarring or pounding.”

“Felt and Asbestos Covering for Steam Surfaces,” Scientific American 357 (December 4, 1880). [353 East 78th is right around the corner from me.  I doubt that many of the residents of this mid-rise apartment building know that an asbestos factory once graced their property.]  See also “The Prevention of Fires in Theaters,” 35 Scientific American 401 (Dec. 23,1876); “Insulated Coverings for Pipes, Boilers, Etc.,” 59 Scientific American 355, 355 (Dec. 8, 1888).

Posted in Historian Testimony, Scientific Evidence | No Comments »

Federal Rules Get a Makeover

December 2nd, 2011

Bellbottoms are out; cuffs are in.  Robert Frost is out; Philip Levine is in.

So too with the Federal Rules.

The Federal Rules of Evidence have been “restyled.” Yesterday, the new, restyled Federal Rules of Evidence went into effect.

A PDF of the new rules is available at several places on the web, including the Federal Evidence Review website, which also has also links to the legislative history and guiding principles for this restyling.   The Legal Information Institute (LII) at Cornell Law School helpfully has posted ebooks, as ePub or mobi files, of the restyled Federal Rules of Civil Procedure, Criminal Procedure, and Evidence.

The legislative history of the restyled Evidence Rules 101-1103 make clear that the changes were designed to make the rules simpler, more readable and understandable, without changing their substantive meaning.  Was this effort worth the time and money?

The rules on expert witness opinion testimony are my particular interest.

Rule 703. Bases of an Expert’s Opinion Testimony

An expert may base an opinion on facts or data in the case that the expert has been made aware of or personally observed. If experts in the particular field would reasonably rely on those kinds of facts or data in forming an opinion on the subject, they need not be admissible for the opinion to be admitted. But if the facts or data would otherwise be inadmissible, the proponent of the opinion may disclose them to the jury only if their probative value in helping the jury evaluate the opinion substantially outweighs their prejudicial effect.

(Legislative History: Pub. L. 93-595, Jan. 2, 1975; Mar. 2, 1987, eff. Oct. 1, 1987; Apr. 17, 2000, eff. Dec. 1, 2000; Apr. 26, 2011, eff. Dec. 1, 2011.)

The rule specifies what happens “[i]f experts in the particular field would reasonably rely on those kinds of facts or data in forming an opinion on the subject,” but what happens “if not“?  The common reading interpolates “only” before “if,” but Rule 703 before and after restyling misses this drafting point.

So too does Rule 702:

Rule 702. Testimony by Expert Witnesses

A witness who is qualified as an expert by knowledge, skill, experience, training, or education may testify in the form of an opinion or otherwise if:

(a) the expert’s scientific, technical, or other specialized knowledge will help the trier of fact to understand the evidence or to determine a fact in issue;

(b) the testimony is based on sufficient facts or data;

(c) the testimony is the product of reliable principles and methods; and

(d) the expert has reliably applied the principles and methods to the facts of the case.

(Legislative History: Pub. L. 93-595, Jan. 2, 1975; Apr. 17, 2000, eff. Dec. 1, 2000; Apr. 26, 2011, eff. Dec. 1, 2011.)

And if not?

The enumeration of (a) through (d) in Rule 702, however, is an improvement for reading and comprehension, especially with the conjunction connecting the last member of the series.

I suppose at age 36, everyone is entitled to a makeover.

Posted in Rule 702, Rule 703, Scientific Evidence | No Comments »

David Michaels’ Public Relations Problem

December 2nd, 2011

OSHA requires strong, credible leadership from someone who will not outrun his scientific headlights, while at the same time enforcing standards that protect workers. President Obama made a serious error in appointing David Michaels, whose scientific and enforcement bona fides are weak.

Michaels has made a career out of targeting industry for perceived ethical lapses, yet he has routinely failed to make adequate disclosures himself, and some of his disclosures are downright deceptive.  This hypocrisy might be shrugged off as part of the politicization of occupational and environmental medicine, except that Michaels is now an Undersecretary of Labor.  When his agency starts handing out legal opinion letters to his former employers in the United States litigation industry, Michaels’ hypocrisy becomes something of a public nuisance and a scandal.  See “Manufacturing Certainty” (Oct. 25, 2011).  The Department of Labor’s “Dear Mr. Wodka” letter can now be found online at OSHA’s website.

Well before David Michaels became head of OSHA, his hypocrisy over conflicts of interest was noteworthy.  See “Hypocrisy In Conflict Disclosure Rules.” In his book, Doubt is Their Product: How Industry’s War on Science Threatens Your Health (2008), Michaels provides no disclosure of his prior activities and testimonial adventures on behalf of the litigation industry.  There is, among his acknowledgments, a tip of the hat to friends and colleagues, such as Steven Wodka.  Wodka is a plaintiffs’ lawyer who retained and paid Michaels in various litigations, but you will not learn that from reading Doubt is Their Product.  Not surprisingly, this book is waved around by plaintiffs’ counsel in cross-examinations in courtrooms all across the United States.

Michaels does reveal that his organization, The Project on Scientific Knowledge and Public Policy (SKAPP), accepted funding from “the Common Benefit Trust, a fund established pursuant to a court order in the Silicone Gel Breast Implant Products Liability Litigation.”  This revelation is, however, quite misleading.  The “Trust” is a fund for plaintiffs’ counsel in the silicone gel breast implant litigation, which was diverted to help support Michaels, and others who would advocate against evidence-based limitations to expert witness opinions.

Michaels insists that SKAPP accepts only unrestricted funding, but this insistence is also misleading.  Plaintiffs’ counsel could feel safe putting “their” money into the coffers of SKAPP, which was openly committed to undermining the implementation of evidence-based standards for causation opinion testimony in federal and state courts.  If the manufacturing industry, as opposed to the litigation industry, funded a not-for-profit, headed up by one of its testifying expert witnesses, most folks would call this maneuver “money laundering.”  Dirty money is dirty money, regardless whose ox is gored.  See also David Michaels & Celeste Monforton, “Scientific Evidence and the Regulatory System: Manufacturing Uncertainty and the Demise of the Formal Regulatory System,” 18 J. Law & Policy 17 (2005) (“Major support for SKAPP is provided by the Common Benefit Trust, a fund established pursuant to a court order in the Silicone Gel Breast Implant Products Liability Litigation.”).

Other anemic or absent conflict of interest disclosures abound in Michaels’ publications.

Michaels has been involved in at least four different mass tort litigations, involving alleged injuries from exposures to asbestos, ortho-toluidene, beryllium, and vinyl chloride.  He has collaborated with Wodka in three of these litigations, by serving as Wodka’s expert witness.  This litigation collaboration should raise serious questions about the “Dear Mr. Wodka letter.”

Asbestos Litigation

Michaels has written several publications about health outcomes in sheet metal workers.  The premise of these papers is that the workers were exposed to asbestos, and they might have greater than expected cancer mortality as a result.  Most of Michaels’ papers fail to reveal that he consulted and testified for asbestos claimants.  See, e.g., David Michaels & Stephen Zoloth, “Asbestos Disease in Sheet Metal Workers: Proportional Mortality Update,” 13 Am. J. Indus. Med. 731-734 (1988).

One of Michaels’ publications on asbestos exposure and health outcomes does contains a disclosure, which even reveals on which side of asbestos litigation he worked:

“This work was supported by the Sheet Metal Occupational Health Institute Trust. Drs. Welch, Michaels, and Dement have worked as consultants for law firms representing individuals with asbestos-related disease. None of the authors have a financial interest in any organization that could profit from the research presented here.”

Laura Welch, Elizabeth Haile, John Dement, and David Michaels, “Change in Prevalence of Asbestos-Related Disease Among Sheet Metal Workers 1986 to 2004,” 131 Chest 863, 863 (2007).  Note the advocacy even in the disclosure.  Law firms that represent only individuals with asbestos-related disease!  Do we infer from this that Michaels did not consult for any law firms that represented individuals who claimed asbestos-related disease, but where the truth in God’s eye would have it that their claims were erroneous?  Perhaps the Principle of Charity requires us to infer that Michaels meant to disclose that he consulted for firms that represented persons claiming asbestos-related disease.  Having read Michaels’ litigation testimony, however, I think he really meant to say that what appears in the article.

There have been many thousands of asbestos cases, most of which have been settled or dismissed.  It is thus difficult to know exactly how many asbestos cases have seen the consulting work of David Michaels.  Clearly, however, some of Michaels’ asbestos testimony was given at the request of Steve Wodka, for Wodka’s clients.  See David Michaels deposition testimony at p. 41,  in Nicastro v. Aceto Corp., New Jersey Superior Court, Law Division for Monmouth County, Docket No. L-3062-08 (Sept. 2, 2009).

Ortho-Toluidine Litigation

According to federal Magistrate Judge H. Kenneth Schroeder, Jr., Steve Wodka represents numerous plaintiffs who claim to have been harmed by exposure to ortho-toluidine.  David Michaels is a common fixture in these cases brought by Wodka.  See Pardee v. E.I. DuPont Nemours & Co., Case 1:07-cv-00268-WMS-HKS Document 29 (W.D.N.Y. March 31, 2008).  Faced with losing his expert witness to OSHA, Wodka noticed a trial deposition de bene esse of David Michaels in several cases.

Michaels was permitted to give his testimony, before moving into his OSHA position, in the following cases:

Pardee v. E.I. DuPont Nemours & Co., W.D.N.Y., Plaintiff, No. 07-CV-0268S(Sr)

Band v. E.I. DuPont Nemours & Co., W.D.N.Y., No. 07-CV-0267S(Sr)

Weist v. E.I. DuPont Nemours & Co., W.D.N.Y., No. 05-CV-0534A(Sr)

Nicastro v. Aceto Corp., New Jersey Superior Court, Law Division for Monmouth County, Docket No. L-3062-08

Polyvinyl Chloride Litigation

David Michaels served as a plaintiffs’ expert witness in at least one PVC case, Lattin v. Borden Chemical Co., New Jersey Superior Court, Law Div. Mercer Cty. Docket No. L-3850-01.  Mr. Wodka was the attorney for plaintiff.

Beryllium Litigation

One of David Michaels’ publications criticized the beryllium industry, on grounds that it advanced weak scientific data and arguments against changes in permissible exposure limits. David Michaels & Celeste Monforton, “Beryllium’s Public Relations Problem: Protecting Workers When There Is No Safe Exposure Level,” 123 Public Health Reports 79 (2008).  In this article, Michaels acknowledges that he “served as an expert witness in a civil suit involving chronic beryllium disease.”  Apparently, Michaels forgot to point out that he was paid for his services, and that the payor was the claimant, whose interests he was advancing in his paper.

Marc Kolanz for one of the companies sued over beryllium health claims noted, in rebuttal, that:

“Dr. Michaels is a paid expert witness in beryllium litigation.  Dr. Michaels’ has not published beryllium industrial hygiene or medical research; however, he has provided litigation support serving as a paid expert witness for plaintiffs in beryllium litigation. Consistent with this role, as a hired advocate for plaintiff’s counsel, he has sought to ‘manufacture certainty’ by applying a hindsight approach to criticize the good works of dedicated beryllium researchers.”

Marc Kolanz, “Beryllium History and Public Policy,” 123 Public Health Reports 423, 427 (2008).

Michaels was an expert witness for Philadelphia plaintiffs’ attorney, Ed Reeves, in the Lonnie Pierce case, in Pennsylvania.

*   *   *   *

There is nothing ignoble or disreputable in serving as an expert witness.  Indeed, real experts may well have an obligation to make their expertise available to the civil and criminal justice system.  What is unseemly is the incessant hypocrisy in accusing manufacturing industry of conflicts of interest, while hiding and misrepresenting litigation industry conflicts.  David Michaels has been in the forefront of this hypocrisy.  The “Dear Mr. Wodka” letter deserves more scrutiny under the principles that Michaels has advocated for manufacturing industry.

Posted in Conflicts of Interest | 1 Comment »

Epidemiology, Risk, and Causation – Report of Workshops

November 15th, 2011

This month’s issue of Preventive Medicine includes a series of papers arising from last year’s workshops on “Epidemiology, Risk, and Causation,” at Cambridge University. The workshops were organized by philosopher Alex Broadbent,  a member of the Department of History and Philosophy of Science, in Cambridge University.  The workshops were financially sponsored by the Foundation for Genomics and Population Health (PHG), a not-for-profit British organization.

Broadbent’s workshops were intended for philosophers of science, statisticians, and epidemiologists, lawyers involved in health effects litigation will find the papers of interest as well.  The themes of workshops included:

  • the nature of epidemiologic causation,
  • the competing claims of observational and experimental research for establishing causation,
  • the role of explanation and prediction in assessing causality,
  • the role of moral values in causal judgments, and
  • the role of statistical and epistemic uncertainty in causal judgments

See Alex Broadbent, ed., “Special Section: Epidemiology, Risk, and Causation,” 53 Preventive Medicine 213-356 (October-November 2011).  Preventive Medicine is published by Elsevier Inc., so you know that the articles are not free.  Still you may want to read these at your local library to determine what may be useful in challenging and defending causal judgments in the courtroom.  One of the interlocutors, Sander Greenland, is of particular interest because he shows up as an expert witness with some regularity.

Here are the individual papers published in this special issue:

Alfredo Morabia, Michael C. Costanza, Philosophy and epidemiology

Alex Broadbent, Conceptual and methodological issues in epidemiology: An overview

Alfredo Morabia, Until the lab takes it away from epidemiology

Nancy Cartwright, Predicting what will happen when we act. What counts for warrant?

Sander Greenland, Null misinterpretation in statistical testing and its impact on health risk assessment

Daniel M. Hausman, How can irregular causal generalizations guide practice

Mark Parascandola, Causes, risks, and probabilities: Probabilistic concepts of causation in chronic disease epidemiology

John Worrall, Causality in medicine: Getting back to the Hill top

Olaf M. Dekkers, On causation in therapeutic research: Observational studies, randomised experiments and instrumental variable analysis

Alexander Bird, The epistemological function of Hill’s criteria

Michael Joffe, The gap between evidence discovery and actual causal relationships

Stephen John, Why the prevention paradox is a paradox, and why we should solve it: A philosophical view

Jonathan Wolff, How should governments respond to the social determinants of health?

Alex Broadbent, What could possibly go wrong? — A heuristic for predicting population health outcomes of interventions, Pages 256-259

Posted in Causation, Scientific Evidence, statistical evidence | No Comments »

The Treatment of Meta-Analysis in the Third Edition of the Reference Manual on Scientific Evidence

November 14th, 2011

Meta-analysis is a statistical procedure for aggregating data and statistics from individual studies into a single summary statistical estimate of the population measurement of interest.  The first meta-analysis is typically attributed to Karl Pearson, circa 1904, who sought a method to overcome the limitations of small sample size and low statistical power.  Statistical methods for meta-analysis, however, did not mature until the 1970s.  Even then, the biomedical scientific community remained skeptical of, if not out rightly hostile to, meta-analysis until relatively recently.

The hostility to meta-analysis, especially in the context of observational epidemiologic studies, was colorfully expressed by Samuel Shapiro and Alvan Feinstein, as late as the 1990s:

“Meta-analysis begins with scientific studies….  [D]ata from these studies are then run through computer models of bewildering complexity which produce results of implausible precision.”

* * * *

“I propose that the meta-analysis of published non-experimental data should be abandoned.”

Samuel Shapiro, “Meta-analysis/Smeta-analysis,” 140 Am. J. Epidem. 771, 777 (1994).  See also Alvan Feinstein, “Meta-Analysis: Statistical Alchemy for the 21st Century,” 48 J. Clin. Epidem. 71 (1995).

The professional skepticism about meta-analysis was reflected in some of the early judicial assessments of meta-analysis in court cases.  In the 1980s and early 1990s, some trial judges erroneously dismissed meta-analysis as a flawed statistical procedure that claimed to make something out of nothing. Allen v. Int’l Bus. Mach. Corp., No. 94-264-LON, 1997 U.S. Dist. LEXIS 8016, at *71–*74 (suggesting that meta-analysis of observational studies was controversial among epidemiologists).

In In re Paoli Railroad Yard PCB Litigation, Judge Robert Kelly excluded plaintiffs’ expert witness Dr. William Nicholson and his testimony based upon his unpublished meta-analysis of health outcomes among PCB-exposed workers.  Judge Kelly found that the meta-analysis was a novel technique, and that Nicholson’s meta-analysis was not peer reviewed.  Furthermore, the meta-analysis assessed health outcomes not experienced by any of the plaintiffs before the trial court.  706 F. Supp. 358, 373 (E.D. Pa. 1988).

The Court of Appeals for the Third Circuit reversed the exclusion of Dr. Nicholson’s testimony, and remanded for reconsideration with instructions.  In re Paoli R.R. Yard PCB Litig., 916 F.2d 829, 856-57 (3d Cir. 1990), cert. denied, 499 U.S. 961 (1991); Hines v. Consol. Rail Corp., 926 F.2d 262, 273 (3d Cir. 1991).  The Circuit noted that meta-analysis was not novel, and that the lack of peer-review was not an automatic disqualification.  Acknowledging that a meta-analysis could be performed poorly using invalid methods, the appellate court directed the trial court to evaluate the validity of Dr. Nicholson’s work on his meta-analysis.

In one of many squirmishes over colorectal cancer claims in asbestos litigation, Judge Sweet in the Southern District of New York was unimpressed by efforts to aggregate data across studies.  Judge Sweet declared that “no matter how many studies yield a positive but statistically insignificant SMR for colorectal cancer, the results remain statistically insignificant. Just as adding a series of zeros together yields yet another zero as the product, adding a series of positive but statistically insignificant SMRs together does not produce a statistically significant pattern.”  In In re Joint E. & S. Dist. Asbestos Litig., 827 F. Supp. 1014, 1042 (S.D.N.Y. 1993).  The plaintiffs’ expert witness who had offered the unreliable testimony, Dr. Steven Markowitz, like Nicholson, another foot soldier in Dr. Irving Selikoff’s litigation machine, did not offer a formal meta-analysis to justify his assessment that multiple non-significant studies, taken together, rule out chance as a likely explanation for an aggregate finding of an increased risk.

Judge Sweet was quite justified in rejecting this back of the envelope, non-quantitative meta-analysis.  His suggestion, however, that multiple non-significant studies could never collectively serve to rule out chance as an explanation for an overall increased rate of disease in the exposed groups is wrong.  Judge Sweet would have better focused on the validity issues in key studies, the presence of bias and confounding, and the completeness of the proffered meta-analysis.  The Second Circuit reversed the entry of summary judgment, and remanded the colorectal cancer claim for trial.  52 F.3d 1124 (2d Cir. 1995).  Over a decade later, with even more accumulated studies and data, the Institute of Medicine found the evidence for asbestos plaintiffs’ colorectal cancer claims to be scientifically insufficient.  Institute of Medicine, Asbestos: Selected Cancers (Wash. D.C. 2006).

Courts continue to go astray with an erroneous belief that multiple studies, all without statistically significant results, cannot yield a statistically significant summary estimate of increased risk.  See, e.g., Baker v. Chevron USA, Inc., 2010 WL 99272, *14-15 (S.D.Ohio 2010) (addressing a meta-analysis by Dr. Infante on multiple myeloma outcomes in studies of benzene-exposed workers).  There were many sound objections to Infante’s meta-analysis, but the suggestion that multiple studies without statistical significance could not yield a summary estimate of risk with statistical significance was not one of them.

In the last two decades, meta-analysis has emerged as an important technique for addressing random variation in studies, as well as some of the limitations of frequentist statistical methods.  In 1980s, articles reporting meta-analyses were rare to non-existent.  In 2009, there were over 2,300 articles with “meta-analysis” in their title, or in their keywords, indexed in the PubMed database of the National Library of Medicine.  See Michael O. Finkelstein and Bruce Levin, “Meta-Analysis of ‘Sparse’ Data: Perspectives from the Avandia Cases” (2011) (forthcoming in Jurimetrics).

The techniques for aggregating data have been studied, refined, and employed extensively in thousands of methods and application papers in the last decade. Consensus guideline papers have been published for meta-analyses of clinical trials as well as observational studies.  See Donna Stroup, et al., “Meta-analysis of Observational Studies in Epidemiology: A Proposal for Reporting,” 283 J. Am. Med. Ass’n 2008 (2000) (MOOSE statement); David Moher, Deborah Cook, Susan Eastwood, Ingram Olkin, Drummond Rennie, and Donna Stroup, “Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement,” 354 Lancet 1896 (1999).  See also Jesse Berlin & Carin Kim, “The Use of Meta-Analysis in Pharmacoepidemiology,” in Brian Strom, ed., Pharmacoepidemiology 681, 683–84 (4th ed. 2005); Zachary Gerbarg & Ralph Horwitz, “Resolving Conflicting Clinical Trials: Guidelines for Meta-Analysis,” 41 J. Clin. Epidemiol. 503 (1988).

Meta-analyses, of observational studies and of randomized clinical trials, routinely are relied upon by expert witnesses in pharmaceutical and so-called toxic tort litigation. Id. See also In re Bextra and Celebrex Marketing Sales Practices and Prod. Liab. Litig., 524 F. Supp. 2d 1166, 1174, 1184 (N.D. Cal. 2007) (holding that reliance upon “[a] meta-analysis of all available published and unpublished randomized clinical trials” was reasonable and appropriate, and criticizing the expert witnesses who urged the complete rejection of meta-analysis of observational studies)

The second edition of the Reference Manual on Scientific Evidence gave very little attention to meta-analysis.  With this historical backdrop, it is interesting to see what the new third edition provides for guidance to the federal judiciary on this important topic.

STATISTICS CHAPTER

The statistics chapter of the third edition gives continues to give scant attention to meta-analysis.  The chapter notes, in a footnote, that there are formal procedures for aggregating data across studies, and that the power of the aggregated data will exceed the power of the individual, included studies.  The footnote then cautions that meta-analytic procedures “have their own weakness,” without detailing what that one weakness is.  RMSE 3d at 254 n. 107.

The glossary at the end of the statistics chapter offers a definition of meta-analysis:

“meta-analysis. Attempts to combine information from all studies on a certain topic. For example, in the epidemiological context, a meta-analysis may attempt to provide a summary odds ratio and confidence interval for the effect of a certain exposure on a certain disease.”

Id. at 289.

This definition is inaccurate in ways that could yield serious mischief.  Virtually all meta-analyses are built upon a systematic review that sets out to collect all available studies on a research issue of interest.  It is a rare meta-analysis, however, that includes “all” studies in its quantitative analysis.  The meta-analytic process involves a pre-specification of inclusionary and exclusionary criteria for the quantitative analysis of the summary estimate of risk.  Those criteria may limit the quantitative analysis to randomized trials, or to analytical epidemiologic studies.  Furthermore, meta-analyses frequently and appropriately have pre-specified exclusionary criteria that relate to study design or quality.

On a more technical note, the offered definition suggests that the summary estimate of risk will be an odds ratio, which may or may not be true.  Meta-analyses of risk ratios may yield summary estimates of risk in terms of relative risk or hazard ratios, or even of risk differences.  The meta-analysis may combine data of means rather than proportions as well.

EPIDEMIOLOGY CHAPTER

The chapter on epidemiology delves into meta-analysis in greater detail than the statistics chapter, and offers apparently inconsistent advice.  The overall gist of the chapter, however, can perhaps best be summarized by the definition offered in this chapter’s glossary:

“meta-analysis. A technique used to combine the results of several studies to enhance the precision of the estimate of the effect size and reduce the plausibility that the association found is due to random sampling error.  Meta-analysis is best suited to pooling results from randomly controlled experimental studies, but if carefully performed, it also may be useful for observational studies.”

Reference Guide on Epidemiology, RSME3d at 624.  See also id. at 581 n. 89 (“Meta-analysis is better suited to combining results from randomly controlled experimental studies, but if carefully performed it may also be helpful for observational studies, such as those in the epidemiologic field.”).  The epidemiology chapter appropriately notes that meta-analysis can help address concerns over random error in small studies.  Id. at 579; see also id. at 607 n. 171.

Having told us that properly conducted meta-analyses of observational studies can be helpful, the chapter hedges considerably:

“Meta-analysis is most appropriate when used in pooling randomized experimental trials, because the studies included in the meta-analysis share the most significant methodological characteristics, in particular, use of randomized assignment of subjects to different exposure groups. However, often one is confronted with nonrandomized observational studies of the effects of possible toxic substances or agents. A method for summarizing such studies is greatly needed, but when meta-analysis is applied to observational studies – either case-control or cohort – it becomes more controversial.174 The reason for this is that often methodological differences among studies are much more pronounced than they are in randomized trials. Hence, the justification for pooling the results and deriving a single estimate of risk, for example, is problematic.175“

Id. at 607.  The stated objection to pooling results for observational studies is certainly correct, but many research topics have sufficient studies available to allow for appropriate selectivity in framing inclusionary and exclusionary criteria to address the objection.  The chapter goes on to credit the critics of meta-analyses of observational studies.  As they did in the second edition of the RSME, the authors repeat their cites to, and quotes from, early papers by John Bailar, who was then critical of such meta-analyses:

“Much has been written about meta-analysis recently and some experts consider the problems of meta-analysis to outweigh the benefits at the present time. For example, John Bailar has observed:

‘[P]roblems have been so frequent and so deep, and overstatements of the strength of conclusions so extreme, that one might well conclude there is something seriously and fundamentally wrong with the method. For the present . . . I still prefer the thoughtful, old-fashioned review of the literature by a knowledgeable expert who explains and defends the judgments that are presented. We have not yet reached a stage where these judgments can be passed on, even in part, to a formalized process such as meta-analysis.’

John Bailar, “Assessing Assessments,” 277 Science 528, 529 (1997).”

Id. at 607 n.177.  Bailar’s subjective preference for “old-fashioned” reviews, which often cherry picked the included studies is, well, “old fashioned.”  More to the point, it is questionable science, and a distinctly minority viewpoint in the light of substantial improvements in the conduct and reporting of meta-analyses of observational studies.  Bailar may be correct that some meta-analyses should have never left the protocol stage, but the RMSE 3d fails to provide the judiciary with the tools to appreciate the distinction between good and bad meta-analyses.

This categorical rejection, cited with apparent approval, is amplified by a recitation of some real or apparent problems with meta-analyses of observational studies.  What is missing is a discussion of how many of these problems can be and are dealt with in contemporary practice:

“A number of problems and issues arise in meta-analysis. Should only published papers be included in the meta-analysis, or should any available studies be used, even if they have not been peer reviewed? Can the results of the meta-analysis itself be reproduced by other analysts? When there are several meta-analyses of a given relationship, why do the results of different meta-analyses often disagree? The appeal of a meta-analysis is that it generates a single estimate of risk (along with an associated confidence interval), but this strength can also be a weakness, and may lead to a false sense of security regarding the certainty of the estimate. A key issue is the matter of heterogeneity of results among the studies being summarized.  If there is more variance among study results than one would expect by chance, this creates further uncertainty about the summary measure from the meta-analysis. Such differences can arise from variations in study quality, or in study populations or in study designs. Such differences in results make it harder to trust a single estimate of effect; the reasons for such differences need at least to be acknowledged and, if possible, explained.176 People often tend to have an inordinate belief in the validity of the findings when a single number is attached to them, and many of the difficulties that may arise in conducting a meta-analysis, especially of observational studies such as epidemiologic ones, may consequently be overlooked.177“

Id. at 608.  The authors are entitled to their opinion, but their discussion leaves the judiciary uninformed about current practice, and best practices, in epidemiology.  A categorical rejection of meta-analyses of observational studies is at odds with the chapter’s own claim that such meta-analyses can be helpful if properly performed.  What was needed, and is missing, is a meaningful discussion to help the judiciary determine whether a meta-analysis of observational studies was properly performed.

MEDICAL TESTIMONY CHAPTER

The chapter on medical testimony is the third pass at meta-analysis in RMSE 3d.   The second edition’s chapter on medical testimony ignored meta-analysis completely; the new edition addresses meta-analysis in the context of the hierarchy of study designs:

“Other circumstances that set the stage for an intense focus on medical evidence included

(1) the development of medical research, including randomized controlled trials and other observational study designs;

(2) the growth of diagnostic and therapeutic interventions;141

(3) interest in understanding medical decision making and how physicians reason;142 and

(4) the acceptance of meta-analysis as a method to combine data from multiple randomized trials.143“

RMSE 3d at 722-23.

The chapter curiously omits observational studies, but the footnote reference (note 143) then inconsistently discusses two meta-analyses of observational, rather than experimental, studies:

“143. … Video Software Dealers Ass’n v. Schwarzenegger, 556 F.3d 950, 963 (9th Cir. 2009) (analyzing a meta-analysis of studies on video games and adolescent behavior); Kennecott Greens Creek Min. Co. v. Mine Safety & Health Admin., 476 F.3d 946, 953 (D.C. Cir. 2007) (reviewing the Mine Safety and Health Administration’s reliance on epidemiological studies and two meta-analyses).”

Id. at 723 n.143.

The medical testimony chapter then provides further confusion by giving a more detailed listing of the hierarchy of medical evidence in the form of different study designs:

“3. Hierarchy of medical evidence

With the explosion of available medical evidence, increased emphasis has been placed on assembling, evaluating, and interpreting medical research evidence.  A fundamental principle of evidence-based medicine (see also Section IV.C.5, infra) is that the strength of medical evidence supporting a therapy or strategy is hierarchical.  When ordered from strongest to weakest, systematic review of randomized trials (meta-analysis) is at the top, followed by single randomized trials, systematic reviews of observational studies, single observational studies, physiological studies, and unsystematic clinical observations.150 An analysis of the frequency with which various study designs are cited by others provides empirical evidence supporting the influence of meta-analysis followed by randomized controlled trials in the medical evidence hierarchy.151 Although they are at the bottom of the evidence hierarchy, unsystematic clinical observations or case reports may be the first signals of adverse events or associations that are later confirmed with larger or controlled epidemiological studies (e.g., aplastic anemia caused by chloramphenicol,152 or lung cancer caused by asbestos153). Nonetheless, subsequent studies may not confirm initial reports (e.g., the putative association between coffee consumption and pancreatic cancer).154“

Id. at 723-24.  This discussion further muddies the water by using a parenthetical to suggest that meta-analyses of randomized clinical trials are equivalent to systematic reviews of such studies — “systematic review of randomized trials (meta-analysis).” Of course, systematic reviews are not meta-analyses, although they are a necessary precondition for conducting a meta-analysis.  The relationship between the procedures for a systematic review and a meta-analysis are in need of clarification, but the judiciary will not find it in the new Reference Manual.

Posted in Causation, Rule 702, Scientific Evidence, statistical evidence | 1 Comment »

OSHA’s HazCom Standard — Statistical and Scientific Nonsense

November 13th, 2011

Almost 28 years ago, the United States Department of Labor (Occupational Safety and Health Administration or OSHA) promulgated The Hazard Communication Standard. 29 C.F.R. § 1910.1200 (November 1983; effective date November 25, 1985) (HazCom standard).  Initially the HazCom standard applied to importers and manufacturers of chemicals.  Starting one year later, November 25, 1986, the standard covered manufacturing employers, under OSHA jurisdiction, by defining their duties to protect and inform employees.

The HazCom standard applies to all chemical manufacturers and distributors and to

“any chemical which is known to be present in the workplace in such a manner that employees may be exposed under normal conditions of use or in a foreseeable emergency.”

29 C.F.R. § 1910.1200(b)(1), and (b)(2).  The standard requires manufacturers and distributors of hazardous chemicals inform not only their own employees of the dangers posed by the chemicals, but downstream employers and employees as well.  The standard implements this duty to warn downstream employers’ employees by requiring that containers of hazardous chemicals leaving the workplace are labeled with “appropriate hazard warnings.”  See Martin v. American Cyanamid Co., 5 F.3d 140, 141-42 (6th Cir. 1993) (reviewing agency’s interpretation of the standard).

The HazCom standard attempts to provide some definition of the health hazards for which warnings are required:

“For health hazards, evidence which is statistically significant and which is based on at least one positive study conducted in accordance with established scientific principles is considered to be sufficient to establish a hazardous effect if the results of the study meet the definitions of health hazards in this section.”

29 C.F.R. § 1910.1200(d)(2).

This regulatory language is troubling. What does statistically significant mean?  The concept remains important in health effects research, but several writers have subjected the use of significance testing specifically, and frequentist statistics generally, to criticisms.  See, e.g., Stephen T. Ziliak and Deirdre N. McCloskey, The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice, and Lives (Ann Arbor 2008) (example of one of the more fringe, and not particularly cogent, criticisms of frequentist statistics).  And what are the “established scientific principles,” which would allow a single “positive study” to “establish” a hazardous “effect”?

The HazCom standard is important not only for purposes of regulatory compliance, but for its potential implications for products liability law, as well.  With its importance in mind, what can be said about the definition of health hazard, provided in 29 C.F.R. § 1910.1200(d)(2)?

Perhaps a good place to start is with the guidance provided by OSHA on compliance with the HazCom standard.  To be sure, like most agency guidance statements, this one is prefaced with caveats and cautions:

“This guidance is not a standard or regulation, and it creates no new legal obligations. It is advisory in nature, informational in content, and is intended to assist employers in providing a safe and healthful workplace. Pursuant to the Occupational Safety and Health Act, employers must comply with safety and health standards promulgated by OSHA or by a state with an OSHA-approved state plan. In addition, pursuant to Section 5(a)(1), the General Duty Clause of the Act, employers must provide their employees with a workplace free from recognized hazards likely to cause death or serious physical harm. Employers can be cited for violating the General Duty Clause if there is a recognized hazard and they do not take reasonable steps to prevent or abate the hazard. However, failure to implement any specific recommendations in this guidance is not, in itself, a violation of the General Duty Clause. Citations can only be based on standards, regulations, and the General Duty Clause.”

U.S. Dep’t of Labor, Guidance for Hazard Determination for Compliance with the OSHA Hazard Communication Standard (29 CFR § 1910.1200) (July 6, 2007).

Section II of the Guidance describes how manufacturers may assess whether their chemicals are “hazardous.”  A health hazard is defined as a chemical

“for which there is statistically significant evidence based on at least one study conducted in accordance with established scientific principles that acute or chronic health effects may occur in exposed employees.”

A fair-minded person might object that this is no guidance at all.  Statistically significant is not defined in the regulations. Study is not defined.  The guidance specifies that the study or studies must be conducted in accordance with “established scientific principles,” but must the interpretation or judgment of causality be made similarly in accordance with such principles? One would hope so, but the Guidance does not really specify.  The use of “may” seems to inject a level of conjecture or speculation into the hazard assessment.

Section V of the Guidance addresses data analysis, and here the agency attempts to provide some meaning to statistical significance and other terms in the regulation, but in doing so, the Guidance offers incoherent, incredible advice.

The Guidance notes that the regulation specifies one “positive study,” which presumably is a study that is some evidence in favor of an “effect.”  Because we are dealing with chemical exposures in occupational settings, the studies at issue will be, at best, observational studies.  Randomized clinical trials are out.  The one study (at least) at issue must be sufficient to establish a hazardous effect if that effect is considered a “health hazard” within the meaning of the regulations.  This is problematic on many levels.  What sort of study are we discussing?  An experimental study in planaria worms, a case study of a single human, an ecological study, or an analytical epidemiologic (case-control or cohort) study?  Whatever the study is, it would be a most remarkable study if it alone were “sufficient” to “establish” an “effect.”

A reasonable manufacturer or disinterested administrator surely would interpret the sufficiency requirement to mean that the entire evidentiary display must be considered rather than whether one study, taken in isolation, ripped from its scientific context, should be used to suggest a duty to warn.  The Guidance, and the regulations, however, never address the real-world complexity of hazard assessment.

Section V of the Guidance offers a failed attempt to illuminate the meaning of statistical significance:

“Statistical significance is a mathematical determination of the confidence in the outcome of a test. The usual criterion for establishing statistical significance is the p-value (probability value). A statistically significant difference in results is generally indicated by p < 0.05, meaning there is less than a 5% probability that the toxic effects observed were due to chance and were not caused by the chemical. Another way of looking at it is that there is a 95% probability that the effect is real, i.e., the effect seen was the result of the chemical exposure.”

Few statisticians or scientists would accept the proffered definition as acceptable.  The Guidance’s statement that a p-value is equivalent to the probability of the “toxic effect” occurring by chance is unacceptable for several reasons.

First, it is a notoriously incorrect, fallacious statement of the meaning of a p-value:

“Since p is calculated by assuming the null hypothesis is correct (that there is no difference [between observed and expected] in the full population), the p-value cannot give the chance that this hypothesis is true.  The p-value merely gives the chance of getting evidence against the null hypothesis as strong or stronger than the evidence at hand — assuming that the null hypothesis … is correct.”

David H. Kaye, David E. Bernstein, and Jennifer L. Mnookin, The New Wigmore: Expert Evidence § 12.8.2, at 559 (2d ed. 2010) (discussing the transpositional fallacy).

Second, even if we could ignore the statistical solecism, the Guidance’s use of a mechanical test for statistical significance is troubling.  The p-value is not necessarily an appropriate protection against Type I error, or a “false alarm” that there is an association between the exposure and outcome of interest.  Multiple testing and other aspects of a study may inflate the number of false alarms to the point that a study with a low p-value, even one much lower than 5%, will not rule out the likely role of chance as an explanation for the study’s result.

Third, the Guidance’s suggestion that “statistical significance” boils down to a conclusion that the “effect is real” may be its greatest offense against scientific and statistical methodology.  Section V of the Guidance emphasizes that the HazCom standard states that

“evidence that is statistically significant and which is based on at least one positive study conducted in accordance with established scientific principles is considered to be sufficient to establish a hazardous effect if the results of the study meet the [HCS] definitions of health hazards.”

This is nothing more than semantic fiat and legerdemain.

Statistical significance may, in some circumstances, permit an inference that the divergence from the expected was not likely due to chance, but it cannot, in the context of observational studies, allow for a conclusion that the divergence resulted because of a cause-effect relationship between the exposure and the outcome.  Statistical significance cannot rule out systemic bias or confounding in the study; nor can it help us reconcile inconsistencies across studies.  The study may have identified an association, which must be assessed for its causal or non-causal nature, in the context of all relevant evidence.  See Arthur Bradford Hill, “The Environment and Disease: Association or Causation?” 58 Proc. Royal Soc’y Med. 295 (1965).”

The OSHA Guidance is really no guidance at all.  Ensuring worker health and safety by requiring employers to provide industrial hygiene protections for workers is an exceedingly important task, but this aspect of the HazCom standard is incoherent and incompetent. Workers and employers are in the dark, and product suppliers are vulnerable to arbitrary and capricious enforcement.

Posted in statistical evidence | No Comments »

Lording the Data – Scientific Fraud

November 10th, 2011

Last week, the New York Times published a news story about psychologist Diederik Stapel, of the Netherlands.  Tilburg University accused him of having committed research fraud  in several dozen published papers, including the journal Science, the official journal of the AAAS.  See Benedict Carey, “Fraud Case Seen as a Red Flag for Psychology Research: Noted Dutch Psychologist, Stapel, Accused of Research Fraud,” New York Times (Nov. 2, 2011).  The Times expressed surprise over the suggestion that psychology is plagued by fraud and sloppy research.  The surprise is that there are not more stories in the lay media over the poor quality of scientific research.  The readers of Retraction Watch, and the Office of Research Integrity’s blog will recognize how commonplace Stapel’s fraud is.

Stapel’s fraud has wide-ranging implications for the doctoral students, whose dissertations he supervised, and for colleagues, with whom he collaborated.  Stapel apologized and expressed his regret, but his conduct leaves a large body of his work, and that of others, under a cloud of suspicion.

Lording the Data

The University committee reported that Stapel had escaped detection for a long time because he was “lord of the data,” by refusing to disclose and share the data.

“Outright fraud may be rare, these experts say, but they contend that Dr. Stapel took advantage of a system that allows researchers to operate in near secrecy and massage data to find what they want to find, without much fear of being challenged.”

Benedict Carey, “Fraud Case,” New York Times (Nov. 2, 2011).  Data sharing is preached but rarely practice.

In a recent publication, Dr. Wicherts and his colleagues, at the University of Amsterdam, reported that two-thirds of his sample of Dutch research psychologists refused to share their data, in contravention of the established ethical rules of the discipline. Remarkably, many of the refuseniks had explicit contractual obligations with their publishing journals to provide data.  Jelte Wicherts, Marjan Bakker, Dylan Molenaar, “Willingness to Share Research Data Is Related to the Strength of the Evidence and the Quality of Reporting of Statistical Results,” PLoS ONE 6(11): e26828 (Nov. 2, 2011)

Scientific fraud seems no more common among scientists with industry ties, which are so often the subject of ad hominem conflict of interest claims.  Instead, fraudfeasors such as Stapel or Hwang Woo-suk are more often simply egotistical, narcissistic, self-aggrandizing, self-promoting, or delusional.  In the United States, litigation, occasionally has brought out charlatans, but it has also resulted in high-quality studies that have provided strong evidence for or against litigation claims.  Compare Hon. Jack B. Weinstein, “Preliminary Reflections on Administration of Complex Litigation” 2009 Cardozo L. Rev. de novo 1, 14 (2009) (describing plaintiffs’ expert witnesses in silicone litigation as “charlatans” and the litigation as largely based upon fraud) with Committee on the Safety of Silicone Breast Implants, Institute of Medicine, Safety of Silicone Breast Implants (Wash. D.C. 1999) (reviewing studies, many of which were commissioned by litigation defendants, and which collectively showed lack of association between silicone and autoimmune diseases).

The relation between litigation and research is one that has typically been approached by self-righteous voices, such as David Michaels and David Egilman, and others who have their own deep conflicts of interest.  What is clear is that all litigants, as well as the public, would benefit from enforcing data sharing requirements.  See “Litigation and Research” (April 15, 2007) (science should not be built upon blind trust of scientists: “Nullius in verba.”).

The Times article emphasized Wicherts’ research about lack of data sharing, and suggested that data sharing could improve the quality of scientific publications.  The time may have come, however, for sterner measures of civil and criminal penalties for scientists who abuse and waste governmental funding, or who aid and abet fraudulent litigation.

Posted in Conflicts of Interest, Data Sharing, Scientific Evidence, Underlying Data | No Comments »

New-Age Levellers – Flattening Hierarchy of Evidence

October 30th, 2011

The Levelers were political dissidents in England, in the middle of the 17th century.  Among their causes, Levelers advanced popular sovereignty, equal protection of the law, and religious tolerance.

The political agenda of the Levelers sounds quite noble to 21st century Americans, but their ideals have no place in the world of science:  not all opinions or scientific studies are created equally; not all opinions are worthy of being taken seriously in scientific discourse or in courtroom presentations of science; and not all opinions should be tolerated, especially when they claim causal conclusions based upon shoddy or inadequate evidence.

In some litigations, legal counsel set out to obscure the important quantitative and qualitative distinctions among scientific studies.  Sometimes, lawyers find cooperative expert witnesses, willing to engage in hand waving about “the weight of the evidence,” where the weights are assigned post hoc, in a highly biased fashion.  No study (that favors the claim) left behind.  This is not science, and it is not how science operates, even though some expert witnesses, such as Professor Cranor in the Milward case, have been able to pass off their views as representative of scientific practice.

A sound appreciation of how scientists evaluate studies, and of why not all studies are equal, is essential to any educated evaluation of scientific controversies.  Litigants who face high-quality studies, with results inconsistent with their litigation claims, may well resort to “leveling” of studies.  This leveling may be advanced out of ignorance, but more likely the leveling is an attempt to snooker courts with evidence from exploratory, preliminary, and hypothesis-generating studies as somehow equal to, or greater than, the value of hypothesis-testing studies.

Some of the leveling tactics that have become commonplace in litigation include asserting that:

  • All experts witnesses are the same;
  • All expert witnesses conduct the same analysis;
  • All expert witnesses read articles, interpret them, and offer opinions;
  • All expert witnesses are inherently biased;
  • All expert witnesses select the articles to read and interpret in line with their biases;
  • All epidemiologic studies are the same;
  • All studies are flawed; and
  • All opinions are, in the final analysis, subjective.

This leveling strategy can be seen in Professor Margaret Berger’s introduction to the Reference Manual on Scientific Evidence (RMSE 3d), where she supported an ill-defined “weight-of-the-evidence” approach to causal judgments. See “Late Professor Berger’s Introduction to the Reference Manual on Scientific Evidence” (Oct. 23, 2011).

Other chapters in the RMSE 3d are at odds with Berger’s introduction.  The epidemiology chapter does not explicitly address the hierarchy of studies, but it does describe cross-sectional, ecological, and secular trend studies are less able to support causal conclusions.  Cross-sectional studies are described as “rarely useful in identifying toxic agents,” RMSE 3d at 556, and as “used infrequently when the exposure of interest is an environmental toxic agent,” RMSE 3d at 561.  Cross-sectional studies are described as hypothesis-generating as opposed to hypothesis testing, although not in those specific terms.  Id. (describing cross-sectional studies as providing valuable leads for future research).  Ecological studies are described as useful for identifying associations, but not helpful in determining whether such associations are causal; and ecological studies are identified as a fertile source of error in the form of the “ecological fallacy.”  Id. at 561 -62.

The epidemiology chapter perhaps weakens its helpful description of the limited role of ecological studies by citing, with apparent approval, a district court that blinked at its gatekeeping responsibility to ensure that testifying expert witnesses did, in fact, rely upon “sufficient facts or data,” as well as upon studies that are “of a type reasonably relied upon by experts in the particular field in forming opinions or inferences upon the subject.” Rule 703. RMSE 3d at 561 n.34 (citing Cook v. Rockwell International Corp., 580 F. Supp. 2d 1071, 1095–96 (D. Colo. 2006), where the district court acknowledged the severe limitations of ecological studies in supporting causal inferences, but opined that the limitations went to the weight of the study). Of course, the insubstantial weight of an ecological study is precisely what may result in the study’s failure to support a causal claim.

The ray of clarity in the epidemiology chapter about the hierarchical nature of studies is muddled by an attempt to level epidemiology and toxicology.  The chapter suggests that there is no hierarchy of disciplines (as opposed to studies within a discipline).  RMSE 3d at 564 & n.48 (citing and quoting symposium paper that “[t]here should be no hierarchy [among different types of scientific methods to determine cancer causation]. Epidemiology, animal, tissue culture and molecular pathology should be seen as integrating evidences in the determination of human carcinogenicity.” Michele Carbone et al., “Modern Criteria to Establish Human Cancer Etiology,” 64 Cancer Res. 5518, 5522 (2004).)  Carbone, of course, is best known for his advocacy of a viral cause (SV40), of human mesothelioma, a claim unsupported, and indeed contradicted, by epidemiologic studies.  His statement does not support the chapter’s leveling of epidemiology and toxicology, and Carbone is, in any event, an unlikely source to cite.

The epidemiology chapter undermines its own description of the role of study design in evaluating causality by pejoratively asserting that most epidemiologic studies are “flawed”:

“It is important to emphasize that all studies have ‘flaws’ in the sense of limitations that add uncertainty about the proper interpretation of the results.9 Some flaws are inevitable given the limits of technology, resources, the ability and willingness of persons to participate in a study, and ethical constraints. In evaluating epidemiologic evidence, the key questions, then, are the extent to which a study’s limitations compromise its findings and permit inferences about causation.”

RSME 3d at 553.  This statement is actually a significant improvement over the second edition, where the authors of the epidemiology chapter asserted, without qualification:

“It is important to emphasize that most studies have flaws.”

RMSE 2d 337.  The “flaws” language from the earlier chapter was used on occasion by courts that were set on ignoring competing interpretations of epidemiologic studies.  Since all or most studies are flawed, why bother figuring out what is valid and reliable?  Just let the jury sort it out.  This is not an aid to gatekeeping, but rather a prescription for allowing the gatekeeper to call in sick.

The current epidemiology chapter essentially backtracks from the harsh connotations of its use of the term “flaws,” by now equating the term with “limitations.”  Flaws and limitations, however, are quite different from one another.  What is left out in the third edition’s description is the sense that there are indeed some studies that are so flawed that they must be disregarded altogether.  There may also be limitations in studies, especially observational studies, which is why the party with the burden of proof should generally not be allowed to proceed with only one or two epidemiologic studies.  Rule 702, after all, requires that an expert opinion to be based upon “sufficient facts or data.”

The RSME 3d chapter on medical evidence is a refreshing break from the leveling approach seen elsewhere.  Here at least, the chapter authors devote several pages to explaining the role of study design in assessing an etiological issue:

“3. Hierarchy of medical evidence

With the explosion of available medical evidence, increased emphasis has been placed on assembling, evaluating, and interpreting medical research evidence.  A fundamental principle of evidence-based medicine (see also Section IV.C.5, infra) is that the strength of medical evidence supporting a therapy or strategy is hierarchical.

When ordered from strongest to weakest, systematic review of randomized trials (meta-analysis) is at the top, followed by single randomized trials, systematic reviews of observational studies, single observational studies, physiological studies, and unsystematic clinical observations.150 An analysis of the frequency with which various study designs are cited by others provides empirical evidence supporting the influence of meta-analysis followed by randomized controlled trials in the medical evidence hierarchy.151 Although they are at the bottom of the evidence hierarchy, unsystematic clinical observations or case reports may be the first signals of adverse events or associations that are later confirmed with larger or controlled epidemiological studies (e.g., aplastic anemia caused by chloramphenicol,152 or lung cancer caused by asbestos153). Nonetheless, subsequent studies may not confirm initial reports (e.g., the putative association between coffee consumption and pancreatic cancer).154“

John B. Wong, Lawrence O. Gostin, and Oscar A. Cabrera, “Reference Guide on Medical Testimony,” RMSE 3d 687, 723 -24 (2011).  The third edition’s chapter is a significant improvement of the second edition’s chapter on medical testimony, which does not mention the hierarchy of evidence.  Mary Sue Henifin, Howard M. Kipen, and Susan R. Poulter, ” Reference Guide on Medical Testimony,” RMSE 2d 440 (2000).  Indeed, the only time the word “hierarchy” appeared in the entire second edition was in connection with the hierarchy of the federal judiciary.

The tension, contradictions, and differing emphases among the various chapters of the RSME 3d point to an important “flaw” in the new edition.  The chapters appear to have been written largely in isolation, and without much regard for what the other chapters contain.  The chapters overlap, and indeed contradict one another on key points.  Witness Berger’s rejection of the hierarchy of evidence, the epidemiology chapter’s inconstant presentation of the concept without mentioning it by name, and the medical testimony chapter’s embrace and explicit presentation of the hierarchical nature of medical study evidence.  Fortunately, the laissez-faire editorial approach allowed the disagreement to remain, without censoring any position, but the federal judiciary is not aided by the contradiction and tension in the approaches.

Given the importance of the concept, even the medical testimony chapter in RSME 3d may seem to be too little, too late to be helpful to the judiciary.  There are book-length treatments of systematic reviews and “evidence-based medicine”: the three pages in Wong’s chapter barely scratch the surface of this important topic of how evidence is categorized, evaluated, and synthesized in making judgments of causality.

There are many textbooks and articles available to judges and lawyers on how to assess medical studies.  Recently, John Cherrie has posted on his blog, OH-world, about a series of 17 articles, in the journal Aerzteblatt International, on the proper evaluation of medical and epidemiologic studies.

These papers, overall, make the point that not all studies are equal, and that not all evidentiary displays are adequate to support conclusions of causal association.  The papers are available without charge from the journal’s website:

01. Critical Appraisal of Scientific Articles

02. Study Design in Medical Research

03. Types of Study in Medical Research

04. Confidence Interval or P-Value?

05. Requirements and Assessment of Laboratory Tests: Inpatient Admission Screening

06. Systematic Literature Reviews and Meta-Analyses

07. The Specification of Statistical Measures and Their Presentation in Tables and Graphs

08. Avoiding Bias in Observational Studies

09. Interpreting Results in 2×2 Tables

10. Judging a Plethora of p-Values: How to Contend With the Problem of Multiple Testing

11. Data Analysis of Epidemiological Studies

12. Choosing statistical tests

13. Sample size calculation in clinical trials

14. Linear regression analysis

15. Survival analysis

16. Concordance analysis

17. Randomized controlled trials

This year, the Journal of Clinical Epidemiology began publishing a series of papers, known by the acronym GRADE, which aim to provide guidance on how studies are categorized and assessed for their evidential quality in supporting treatments and intervention.  The GRADE project is led by Gordon Guyatt, who is known for having coined the term “evidence-based medicine,” and written widely on the subject.  Guyatt, along with his colleagues including Peter Tugwell (who was one of the court-appointed expert witnesses in MDL 926), has described the GRADE project:

“The ‘Grades of Recommendation, Assessment, Development, and Evaluation’ (GRADE) approach provides guidance for rating quality of evidence and grading strength of recommendations in health care. It has important implications for those summarizing evidence for systematic reviews, health technology assessment, and clinical practice guidelines. GRADE provides a systematic and transparent framework for clarifying questions, determining the outcomes of interest, summarizing the evidence that addresses a question, and moving from the evidence to a recommendation or decision. Wide dissemination and use of the GRADE approach, with endorsement from more than 50 organizations worldwide, many highly influential   http://www.gradeworkinggroup.org/), attests to the importance of this work. This article introduces a 20-part series providing guidance for the use of GRADE methodology that will appear in the Journal of Clinical Epidemiology.”

Gordon Guyatt, Andrew D. Oxman, Holger Schünemann, Peter Tugwell, Andre Knottnerus, “GRADE guidelines – new series of articles in Journal of Clinical Epidemiology,” 64 J. Clin. Epidem. 380 (2011).  See also Gordon Guyatt, Andrew Oxman, et al., for the GRADE Working Group, “Rating quality of evidence and strength of recommendations GRADE: an emerging consensus on rating quality of evidence and strength of recommendations,” 336 Brit. Med. J. 924 (2008).  [pdf]

Of the 20 papers planned, 9 of the GRADE papers have been published to date in the Journal of Clinical Epidemiology:

01 Intro – GRADE evidence profiles & summary of findings tables

02 Framing question & deciding on important outcomes

03 Rating quality of evidence

04 Rating quality of evidence – study limitations (risk of bias)

05 Rating the quality of evidence—publication bias

06 Rating up quality of evidence – imprecision

07 Rating quality of evidence – inconsistency

08 Rating quality of evidence – indirectness

09 Rating up quality of evidence

The GRADE guidance papers focus on the efficacy of treatments and interventions, but in doing so, they evaluate “effects” and are thus applicable to the etiologic issues of alleged harm that find their way into court.  The papers build on other grading systems advanced previously by the Oxford Center for Evidence-Based Medicine, the U.S. Preventive Services Task Force (Agency for Healthcare Research and Quality AHRQ), the Cochrane Collaboration, as well as many individual professional organizations.

GRADE has had some success in harmonizing disparate grading systems, and forging a consensus among organizations that had been using their own systems, such as the  World Health Organization, the American College of Physicians, the American Thoracic Society, the Cochrane Collaboration, the American College of Chest Physicians, the British Medical Journal, and Kaiser Permanente.

There are many other important efforts to provide consensus support for improving the quality of the design, conduct, and reporting of published studies, as well as the interpretation of those studies once published.  Although the RSME 3d does a good job of introducing its readers to the basics of study design, it could have done considerably more to help judges become discerning critics of scientific studies and of conclusions based upon individual or multiple studies.

Posted in Rule 702, Scientific Evidence | No Comments »

Historians As Expert Witnesses – A Wiki

October 28th, 2011

“The one duty we owe to history is to rewrite it.”

Oscar Wilde, The Critic As Artist (1891)

“What will history say?  History, sir, will tell lies as usual.”

George Bernard Shaw, The Devil’s Disciple (1901)

* * * * * * * * * * * * * * * * * * * * * * * * *

The Defense Research Institute recently announced that Bill Childs, a professor at the Western New England University School of Law, will be speaking the use of historians as expert witnesses in litigation.  Having puzzled about this very issue in previous writings, I look forward to Professor Childs’ contributions on the issue.  The announcement also noted Professor Childs’ creation, “the Historians as Experts Wiki,” which I knew about, but had not previously visited.

The wiki is a valuable resource of information about historians who have participated in the litigation process in all manner of cases, including art, asbestos, creationism, native Americans, holocaust, products liability, intellectual property, and voting rights.  There are pages for each historian witness, including expert witnesses in other fields, who have given testimony of an explicitly historical nature. The website is still in its formative stages, but it holds great promise as a resource to lawyers who are researching historians who have been listed as expert witnesses in their cases.

Most of my musings about historians as expert witnesses have been provoked by those who have testified about the history of silicosis.  Last year, I presented at a conference sponsored by the International Commission on Occupational Health (ICOH), about such historians.  See “A Walk on the Wild Side,” July 16, 2010.  My presentation abstract, along with all the proceedings of that conference, will be published next year as  “Courting Clio:  Historians and Their Testimony in Products Liability Action,” in: Brian Dolan and Paul Blanc, eds., At Work in the World: Proceedings of the Fourth International Conference on the History of Occupational and Environmental Health, Perspectives in Medical Humanities, University of California Medical Humanities Consortium, University of California Press (2012)(in press).

Posted in Historian Testimony, Rule 702 | No Comments »

« Older Entries
Newer Entries »

The opinions, statements, and asseverations expressed on Tortini are my own, or those of invited guests, and these writings do not necessarily represent the views of clients, friends, or family, even when supported by good and sufficient reason.