TORTINI

For your delectation and delight, desultory dicta on the law of delicts.

Archives »

Counter Cancel Culture – The NAS Conference on Irreproducibility

February 9th, 2020

“The meaning of the world is the separation of wish and fact.”  Kurt Gödel

Back in October 2019, David Randall, the Director of Research, of the National Association of Scholars, contacted me to ask whether I would be interested in presenting at a conference, to be titled “Fixing Science: Practical Solutions for the Irreproducibility Crisis.” David explained that the conference would be aimed at a high level consideration of whether such a crisis existed, and if so, what salutary reforms might be implemented.

As for the character and commitments of the sponsoring organizations, David was candid and forthcoming. I will quote him, without his permission, and ask his forgiveness later:

“The National Association of Scholars is taken to be conservative by many scholars; the Independent Institute is (broadly speaking) in the libertarian camp. The NAS is open to but currently agnostic about the degree of human involvement in climate change. The Independent Institute I take to be institutionally skeptical of consensus climate change theory–e.g., they recently hosted Willie Soon for lecture. A certain number of speakers prefer not to participate in events hosted by institutions with these commitments.”

To me, the ask was for a presentation on how the so-called replication crisis, or the irreproducibility crisis, affected the law. This issue was certainly one I have had much occasion to consider. Although I am aware of the “adjacency” arguments made by some that people should be mindful of whom they align with, I felt that nothing in my participation would compromise my own views or unduly accredit institutional positions of the sponsors.

I was flattered by the invitation, but I did some due diligence on the sponsoring organizations. I vaguely recalled the Independent Institute from my more libertarian days, but the National Association of Scholars (NAS, not to be confused with Nathan A. Schachtman) was relatively unknown to me. A little bit of research showed that the NAS had a legitimate interest in the irreproducibility crisis. David Randall had written a monograph for the organization, which was a nice summary of some of the key problems. The Irreproducibility Crisis of Modern Science: Causes, Consequences,and the Road to Reform (2018).

On other issues, the NAS seemed to live up to its description as “an organization of scholars committed to higher education as the catalyst of American freedom.” I listened to some of the group’s podcasts, Curriculum Vitae, and browsed through its publications. I found myself agreeing with many positions articulated by or through the NAS, and disagreeing with a few positions very strongly.

In looking over the list of other invited speakers, I saw great diversity of view points and approaches, One distinguished speaker, Daniele Fanelli, had criticized the very notion that there was a reproducibility crisis. In the world of statistics, there were strong defenders of statistical tests, and vociferous critics. I decided to accept the invitation, not because I was flattered, but because the replication issue was important, and I believed that I could add something to the discussion before an audience of professional scientists, statisticians, and educated lay persons. In writing to David Randall to accept the invitation, I told him that with respect to the climate change issues, I was not at all put off by healthy skepticism in the face all dogmas. Every dogma will have its day.

I did not give any further consideration to the political aspect of the conference until early January, when I received an email from a scientist, Lenny Teytelman, Ph.D., the C.E.O. of a company protocols.io, which addresses reproducibility issues. Dr Teytelman’s interest in improving reproducibility seemed quite genuine, but he wrote to express his deep concern about the conference and the organizations that were sponsoring it.

Perhaps a bit pedantically, he cautioned me that the NAS was not the National Academy of Sciences, a confusion that never occurred to me because the National Academies has been known as the National Academies of Science, Engineering and Medicine for several years now. Dr. Teytelman’s real concern seemed to be that the NAS is a “‘politically conservative advocacy group’.” (The internal scare quotes were Teytelman’s, but I was not afraid.) According to Dr. Teytelman, the NAS sought to undermine climate science and environmental protection by advancing a call for more reproducible science. He pointed me to what he characterized as an exposé on NAS, in Undark,1 and he cautioned me that the National Association of Scholars’ work is “dangerous.” Finally, Dr. Teytelman urged me to reconsider my decision to participate in the conference.

I did reconsider my decision, but reaffirmed it in an email I sent back to Dr. Teytelman. I realized that I could be wrong, in which case, I would eat my words, confident that they would be most digestible:

Dear Dr Teytelman,

Thank you for your note. I was aware of the piece on Undark’s website, as well as the difference between the NAS and the NASEM. I don’t believe anyone involved in science education would likely to be confused between the two organizations. A couple of years ago, I wrote a teaching module on biomedical causation for the National Academies. This is my first presentation at the request of the NAS, and frankly I am honored by the organization’s request that I present at its conference.

I have read other materials that have been critical of the NAS and its publications on climate change and other issues. I know that there are views of the organization from which I would dissent, but I do not see my disagreement on some issues as a reason not to attend, and present at a conference on an issue of great importance to the legal system.

I am hardly an expert on climate change issues, and that is my failing. Most of my professional work involves health effects regulation and litigation. If the NAS has advanced sophistical arguments against a scientific claim, then the proper antidote will be to demonstrate its fallacious reasoning and misleading marshaling of evidence. I should think, however, as someone interested in improving the reproducibility of scientific research, you will agree that there is much common ground for discussion and reform of scientific practice, on a broader arrange [sic] of issues than climate change.

As for the political ‘conservatism’, of the organization, I am not sure why that is a reason to eschew participation in a conference that should be of great importance to people of all political views. My own politics probably owe much to the influence of Michael Oakeshott, which puts me in perhaps the smallest political tribe of any in the United States. If conservatism means antipathy to post-modernism, identity politics, political orthodoxies, and assaults on Enlightenment values and the Rule of Law, then count me in.

In any event, thanks for your solicitude. I think I can participate and return with my soul intact.

All the best.

Nathan

To his credit, Dr. Teytelman tenaciously continued. He acknowledged that the political leanings of the organizers were not a reason to boycott, but he politely pressed his case. We were now on a first name basis:

Dear Nathan,

I very much applaud all efforts to improve the rigour of our science. The problem here is that this NAS organization has a specific goal – undermining the environmental protection and denying climate change. This is why 7 out of the 21 speakers at the event are climate change deniers. [https://docs.google.com/spreadsheets/d/136FNLtJzACc6_JbbOxjy2urbkDK7GefRZ/edit?usp=sharing] And this isn’t some small fringe effort to be ignored. Efforts of this organization and others like them have now gotten us to the brink of a regulatory change at the United States Environmental Protection Agency which can gut the entire EPA (see a recent editorial against this I co-authored). This conference is not a genuine effort to talk about reproducibility. The reproducibility part is a clever disguise for pushing a climate change denialism agenda.

Best,

Lenny

I looked more carefully at Lenny’s spreadsheet, and considered the issue afresh. We were both pretty stubborn:

Dear Lenny,

Thank you for this information. I will review with interest.

I do not see that the conference is primarily or even secondarily about climate change vel non. There are two scientists, Trafimow and Wasserstein with whom I have some disagreements about statistical methodology. Tony Cox and Stan Young, whatever their political commitments or views on climate change may be, are both very capable statisticians, from whom I have learned a great deal. The conference should be a lively conversation about reproducibility, not about climate change. Given your interests and background, you should go.

I believe that your efforts here are really quite illiberal, although they are in line with the ‘cancel culture’, so popular on campuses these days.

Forty three years ago, I entered a Roman Catholic Church to marry the woman I love. There were no lightning bolts or temblors, even though I was then and I am now an atheist. Yes, I am still married to my first wife. Although I share the late Christopher Hitchins’ low view of the Catholic Church, somehow I managed to overcome my antipathy to being married in what some would call a house of ill repute. I even manage to agree with some Papist opinions, although not for the superstitious ‘reasons’ Papists embrace.

If I could tolerate the RC Church’s dogma for a morning, perhaps you could put aside the dichotomous ‘us and them’ view of the world and participate in what promises to be an interesting conference on reproducibility?

All the best.

Nathan

Lenny kindly acknowledged my having considered his issues, and wrote back a nice note, which I will quote again in full without permission, but with the hope that he will forgive me and even acknowledge that I have given his views an airing in this forum.

Hi Nathan,

We’ll have to agree to disagree. I don’t want to give a veneer of legitimacy to an organization whose goal is not improving reproducibility but derailing EPA and climate science.

Warmly,

Lenny

The business of psychoanalyzing motives and disparaging speakers and conference organizers is a dangerous business for several reasons. First motives can be inscrutable. Second, they can be misinterpreted. And third, they can be mixed. When speaking of organizations, there is the further complication of discerning a corporate motive among the constituent members.

The conference was an exciting, intellectually challenging event, which took place in Oakland, California, on February 7 and 8. I can report back to Lenny that his characterizations of and fears about the conference were unwarranted. While there were some assertions of climate change skepticism made with little or no evidence, the evidence-based presentations essentially affirmed climate change and sought to understand its causes and future course in a scientific way. But climate change was not why I went to this conference. On the more general issue of reform of scientific procedures and methods, we had open debates, some agreement on important principles, and robust and reasoned disagreement.

Lenny, you were correct that the NAS should not be ignored, but you should have gone to the meeting and participated in the conversation.

1 Michael Schulson, “A Remedy for Broken Science, Or an Attempt to Undercut It?” Undark (April 18, 2018).

Posted in Cognitive Biases, Conflicts of Interest, Scientific Evidence, Scientific Misconduct, statistical evidence | Comments Off on Counter Cancel Culture – The NAS Conference on Irreproducibility

Some High-Value Targets for Sander Greenland in 2018

December 27th, 2017

A couple of years ago, Sander Greenland and I had an interesting exchange on Deborah Mayo’s website. I tweaked Sander for his practice of calling out defense expert witnesses for statistical errors, while ignoring whoopers made by plaintiffs’ expert witnesses. See “Significance Levels Made a Whipping Boy on Climate-Change Evidence: Is p < 0.05 Too Strict?” Error Statistics (Jan. 6, 2015).1 Sander acknowledged that he received a biased sample of expert reports through his service as a plaintiffs’ expert witness, but protested that defense counsel avoided him like the plague. In an effort to be helpful, I directed Sander to an example of bad statistical analysis that had been proffered by Dr Bennett Omalu, in a Dursban case, Pritchard v. Dow Agro Sciences, 705 F. Supp. 2d 471 (W.D. Pa. 2010), aff’d, 430 F. App’x 102, 104 (3d Cir. 2011).2

Sander was unimpressed with my example of Dr. Omalu; he found the example “a bit disappointing though because [Omalu] was merely a county medical examiner, and his junk analysis was duly struck. The expert I quoted in my citations was a full professor of biostatistics at a major public university, a Fellow of the American Statistical Association, a holder of large NIH grants, and his analysis (more subtle in its transgressions) was admitted” (emphasis added). Sander expressed an interest in finding “examples involving similarly well-credentialed, professionally accomplished plaintiff experts whose testimony was likewise admitted… .”

Although it was heartening to read Sander’s concurrence in the assessment of Omalu’s analysis as “junk,” Sander’s rejection of Dr. Omalu as merely a low-value target was disappointing, given that Omalu also has a master’s degree in public health, from the University of Pittsburgh, where he claims he studied with Professor Lew Kuller. Omalu has also gained some fame and notoriety for his claim to have identified the problem of chronic traumatic encephalopathy (CTE) among professional football players. After all, even Sander Greenland has not been the subject of a feature-length movie (Concussion), as has Omalu.

I lost track of our exchange in 2015, until recently I was reminded of it when reading an expert report by Professor Martin Wells. Unlike Omalu, Wells meets all the Greenland criteria for high-value targets. He is not only a full, chaired professor but also the statistics department chairman at an ivy-league school, Cornell University. Wells is a fellow of both the American Statistical Association and the Royal Statistical Society, but most important, Wells is a frequent plaintiffs’ expert witness, who is well known to Sander Greenland. Both Wells and Greenland served, side by side, as plaintiffs’ expert witnesses in the pain pump litigation.

So here is the passage in the Wells’ report that is worthy of Greenland’s attention:

“If a 95% confidence interval is specified, the range encompasses the results we would expect 95% of the time if samples for new studies were repeatedly drawn from the same population.”

In re Testosterone Replacement Therapy Prods. Liab. Litig., Declaration of Martin T. Wells, Ph.D., at 2-3 (N.D. Ill., Oct. 30, 2016). Unlike the Dursban litigation involving Bennett Omalu, where the “junk analysis” was excluded, in the litigation against AbbVie for its manufacture and selling of prescription testosterone supplementation, Wells’ opinions were not excluded or limited. In re Testosterone Replacement Therapy Prods. Liab. Litig., No. 14 C 1748, MDL No. 2545, 2017 WL 1833173 (N.D. Ill. May 8, 2017) (denying Rule 702 motions).

Now this statement by Wells surely offends the guidance provided by Greenland and colleagues.3 And it was exactly the sort of misrepresentation that led to a confabulation of the American Statistical Association, and that Association’s consensus statement on statistical significance.4

And here is another example, which occurs not in a distorting litigation forum, but on the pages of an occupational health journal, where the editor in chief, Anthony L. Kiorpes, ranted about the need for better statistical editing and writing in his own journal. See Anthony L Kiorpes, “Lies, damned lies, and statistics,” 33 Toxicol. & Indus. Health 885 (2017). Kiorpes decried he misuse of statistics:

“I am not implying that it is the intent of the scientists who publish in these pages to mislead readers by their use of statistics, but I submit that the misuse of statistics, whether intentional or otherwise, creates confusion and error.”

Id. at 885. Kiorpes then proceeded to hold himself up as Exhibit A to his screed:

“Remember that p values are estimates of the probability that the null hypothesis (no difference) is true.”

Id. Uggh; we seem to be back sliding after the American Statistical Association’s consensus statement.

“Almost all scientists have stated (or have been tempted to state) something like ‘the mean of Group A was greater than that of Group B, but the difference was not statistically significant’. With very few exceptions (which I will mention below), this statement is nonsense.”

* * * * *

“What the statistics are indicating when the p-value is greater than 0.05 is that there is ‘no difference’ between group A and group B.”

Id. at 886.

Let’s hope that this gets Sander Greenland away from his biased sampling of expert witnesses, off the backs of defense expert witnesses, and on to some of the real culprits out there, in the new year.

1 See also “Sander Greenland on ‘The Need for Critical Appraisal of Expert Witnesses in Epidemiology and Statistics’” (Feb. 8, 2015).

2 See also “Pritchard v. Dow Agro – Gatekeeping Exemplified” (Aug. 25, 2014); “Omalu and Science — A Bad Weld” (Oct. 22, 2016); Brian v. Association of Independent Oil Distributors, No. 2011-3413, Westmoreland Cty. Ct. Common Pleas, Order of July 18, 2016 (excluding Dr. Omalu’s testimony on welding and solvents and Parkinson’s disease).

3 See, e.g., Sander Greenland, Stephen J. Senn, Kenneth J. Rothman, John B. Carlin, Charles Poole, Steven N. Goodman, and Douglas G. Altman, “Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations,” 31 Eur. J. Epidem. 337 (2016).

4 Ronald L. Wasserstein & Nicole A. Lazar, “American Statistical Association Statement on statistical significance and p values,” 70 Am. Statistician 129 (2016)

Posted in Cognitive Biases, Conflicts of Interest, Expert Witnesses, Rule 702, statistical evidence | Comments Off on Some High-Value Targets for Sander Greenland in 2018

Alabama Justice and Fundamental Fairness

November 20th, 2017

Judges from Alabama get a bad rap. Like Job, Roy Moore cannot seem to catch a break, despite his professions of great faith. Still Moore ought to be more liberal, given that he, and all of us, are descended from a transgendered woman. After all, Eve was made from a male rib, the tissue of which carried a Y chromosome:

“And the rib, which the LORD God had taken from man, made he a woman, and brought her unto the man.”

Genesis 2:22. Even if Eve magically ended up with two X chromosomes, certainly Judge Moore must accept that the result was part of a cosmic sex change operation. Moore might prefer Lillith, who was built female from the ground up, as his female progenitor.

But Alabama is not so bad, really. My Cousin Vinny got a fair trial for Bill Gambini and Stanley Rothenstein in Beechum County, Alabama. Vinny was surprised by the prosecutor’s generosity in turning over his file, based upon a casual oral request, but as his fiancée Mona Lisa Vito pointed out, the prosecutor had to do this; it’s the law. Brady v. Maryland, 373 U.S. 83 (1963), requires prosecutors to disclose to defense counsel any evidence that could reasonably be construed to favor the defendant.

Actually, Alabama law interpets Brady in the typical fashion to require the prosecutor to disclose only exculpatory evidence. Alabama Rules of Criminal Procedure Rule 16. And Judge Chamberlain Haller, who presided over State v. Gambini, was a stickler for the rules.

And Beechum County prosecutor, Jim Trotter III, was a decent sort. Not only did he offer Vinny his hunting lodge as a quiet place to prepare for trial, Trotter turned over his entire case file upon a casual oral request, without delay. When the evidence appeared to exclupate Gambini and Rothenstein, Trotter withdrew the charges. Outside of Beechum County, that sort of thing happens mostly in movies; in the real world, it is dog eat dog.

The recently publicized case of Wilbert Jones is probably a more typical case. Jones was duly convicted on the testimony of the woman he supposedly raped. The identification, which was the only inculpatory evidence, was shaky, but it was sufficient for one Louisiana jury. What the jury did not hear, however, was that there was another man, who fit the description of the rapist, and who had committed similar crimes in the vicinity. The prosecutor did not think to share that information with Jones’s counsel. When later challenged about the prosecution’s failure to disclose the information, the prosecutors argued that the information was not exculpatory, and would not have made a difference in any event to the “hanging” jury that heard the case. A federal judge, hearing Jones’s petition for the writ of habeas corpus, disagreed, and vacated Jones’s conviction. Jacey Fortin, “Louisiana Man Freed After 45 Years as Conviction is Tossed Out,” N.Y. Times (Nov. 17, 2017). Despite having their conviction vacated, the Louisiana prosecutors have vowed to appeal the decision and retry the case.

Chief Judge Janet DiFiore, of the New York Court of Appeals, is also a stickler for the rules, much like Judge Haller of Beechum County, Alabama. Recently Her Honor issued a new rule that requires trial judges to order, in each case, the prosecution to review their files and disclose all favorable (exculpatory) evidence, at least 30 days before trial. Imagine that; New York prosecutors have to be ordered to comply with the constitutional requirement of disclosure, set out in Brady! See Alan Feuer & James C. McKinley, “Rule Would Push Prosecutors to Release Evidence Favorable to Defense,” N.Y. Times (Nov. 8, 2017); Emmet G. Sullivan, “How New York Courts Are Keeping Prosecutors in Line,” Wall St. J., at A11 (Nov. 18, 2017). See also Andrew Cohen, “Prosecutors Shouldn’t Be Hiding Evidence From Defendants,” The Atlantic (May 13, 2013).

The news media accounts of Chief Judge DiFiore’s newly promulgated rule quotes Innocence Project founder Barry Scheck as stating that the new New York rule is a “big deal.” The New York rule strikes me as a “raw deal,” which leaves to prosecutors to dole out what they think is exculpatory, on the eve of trial. Murder suspects in Beechum County, Alabama, get much better treatment from the likes of Prosecutor Jim Trotter.

The problem, even under the new New York rule, is that prosecutors are advocates and constitutionally incapable of looking at their files from the defense perspective to determine fairly what must be disclosed. Allowing prosecutors to decide what is exculpatory or not is bad policy, bad law, and bad human psychology. The better view would be to require prosecutors to turn over their complete file well in advance of trial, to permit defense counsel to prepare an effective defense.

Criminal trials, like civil trials, end with each side’s lawyer arguing that all the admitted evidence at trial favors his or her side. From the prosecutor’s perspective, none of the evidence exculpates the defendant, or even creates the slightest smidgeon of doubt. How schizophrenic must prosecutors be in order to step inside the psyche of the adversary, before trial, to see the potential inferences and potential for arguments that they will vehemently reject at trial, as utterly implausible and too farfetched to create reasonable doubt?

The entire system of permitting prosecutors to decide the disclosure issue, ex parte, and without supervision, violates the spirit and mandate of Brady. Whatever we may think of Alabama Judge Roy Moore, we could all use some of that Beechum County sense of fair play and due process.

Posted in Cognitive Biases, Trial Procedure | Comments Off on Alabama Justice and Fundamental Fairness

Quackers & Cheese – Trump Picks Kennedy to Study Vaccine Safety

January 11th, 2017

Science necessarily involves a willingness to follow evidence to whatever conclusions are warranted, if conclusions properly can be had. When it comes to vaccination conspiracies, Democrats have it in their political DNA to distrust pharmaceutical companies that research, develop, and manufacture vaccines. The current Republican party, which has been commandeered by theocrats and populists, see vaccination as federal government aggrandizement, and resist vaccination policy as contrary to God’s will. Science is often the loser in the cross-fire.

And so we now have the public spectacle of watching the left and the right join in similar scientific apostasies. Consider how both McCain and Obama both suggested that vaccines and autism were related in the 2008 election. (Although both candidates were to some extent slippery in their suggestions, which might have been appropriate given how little they knew about the controversies.) And consider Michelle Bachmann was converted to a similar view about the HPV vaccine on the basis of a woman’s anecdote about her child. And then on the far left, you have the uplifting story of Robert F. Kennedy Jr, and his brief on how thimerosal supposedly causes autism.

So it should be no surprise that Donald Trump, a Birther, a Mirther, a mid-night Twitterer, should embrace the anti-vaccination movement. Trump has made it clear that he rejects evidence-based policy, and so no one should expect him to embrace a scientific policy that is driven by high-quality scientific evidence. According to Kennedy, Trump wants Kennedy to head up a “commission on vaccine safety and scientific integrity.” Michael D. Shear, Maggie Haberman & Pam Belluckjan, “Anti-Vaccine Activist Says Trump Wants Him to Lead Panel on Immunization Safety,” N.Y. Times (Jan. 10, 2017); Domenico Montanaro, “Despite The Facts, Trump Once Again Embraces Vaccine Skeptics,” National Public Radio (Jan. 10, 2017).

Who needs the National Academy of Medicine when you can put a yutzball lawyer in charge of a “commission”?

Some of the media refer to Robert F. Kennedy Jr. as a vaccine skeptic, but their terminology is grossly inaccurate and misleading. Kennedy is a vaccine denier; he has engaged in a vitriolic campaign against the safety and efficacy of vaccines. He has aligned himself with the most extreme deniers of science, medicine, and public safety, including the likes of Andrew Wakefield and Jenny McCarthy. Kennedy has not merely engaged hyperbolic rhetoric against vaccines, he has used his radio show on the lawsuit industry’s Ring of Fire, to advance his campaign against public health as well as to shill for the lawsuit industry on other issues. See “RFK, Jr.: Science Shows That Autism — Mercury Link Exists – PT. ½,” Ring of Fire (Mar 8, 2011).

Kennedy should not be characterized as a skeptic, when he is a shrill ideologue, for whom science has no method that he is bound to respect. Back in July 2005, Kennedy published an article, “Deadly Immunity,” in both Rolling Stone and on Slate’s website. The article was a hateful screed against Big Pharma and government health agencies for an alleged conspiracy to hide the autism risks of thimerosal preservatives in vaccines. Several years later, on January 16, 2011, Salon retracted the article. See “https://en.wikipedia.org/wiki/Deadly_Immunity” entry in Wikipedia. See also Phil Plait, “Robert F. Kennedy Jr.: Anti-Vaxxer,” Slate (June 5 2013) (describing Kennedy as a full-blown anti-vaccination conspiracy theorist); Rahul K. Parikh, M.D., “Inside the vaccine-and-autism scare: A pediatrician traces the rise of the anti-vaccine movement that falsely linked thimerosal with autism and turned parents away from the most lifesaving medicine in history,” Salon (Sept. 22, 2008); Keith Kloor, “Is Robert F. Kennedy Jr. Anti-Science?” Discover Magazine (June 1, 2013); Steven Novella, “RFK Jr.s Autism Conspiracy Theory,” (Jun 20 2007).

Back in 2008, President Obama apparently considered Robert Kennedy for a cabinet-level position, but on sober reflection, thought better of it. See Steven Novella, “Politics and Science – The RFK Jr. Test,” (Nov. 07 2008). The Wall Street Journal, joined by many others, are now urging Trump to think harder and better about the issue, perhaps with some evidence as well. See Alex Berezow & Hank Campbell, “Ignore Anti-Vaccine Hysteria, Mr. Trump: Robert F. Kennedy Jr.’s conspiracy theories have no place in the White House,” Wall Street J. (Jan. 10, 2017).

Posted in Causation, Cognitive Biases, Conflicts of Interest, Evidence, Public Nuisances, Scientific Evidence, Scientific Misconduct | Comments Off on Quackers & Cheese – Trump Picks Kennedy to Study Vaccine Safety

Art Historian Expert Testimony

August 15th, 2016

Art appraisal and authentication is sometimes held out as a non-technical and non-scientific area of expertise, and as such, not subject to rigorous testing.[1] But to what extent is this simply excuse mongering for an immature field of study? The law has seen way too much of this sort of rationalization in criminal forensic studies.[2] If an entire field of learning suffers from unreliability because of its reliance upon subjective methodologies, lack of rigor, inability or unwillingness to use measurements, failure to eliminate biases through blinding, and the like, then do expert witnesses in this field receive a “pass” under Rule 702, simply because they are doing reasonably well compared with their professional colleagues?

In the movie Who the Fuck is Jackson Pollack, the late Thomas Hoving was interviewed about the authenticity of a painting claimed to have been “painted” by Jackson Pollack. Hoving “authoritatively,” and with his typical flamboyance, averred that the disputed painting was not a Pollack because the work “did not sing to me like a Pollack.” Hoving did not, however, attempt to record the notes he heard; nor did Hoving speak to what key Pollack usually painted in.

In a recent case of defamation and tortious interference with prospective business benefit, a plaintiff sued over the disparagement of a painting’s authenticity and provenance. As a result of the defendants’ statements that the painting at issue was not created by Peter M. Doig, auction houses refused to sell the painting held by plaintiff. In litigation, the plaintiff proffered an expert witness who opined that the painting was, in fact, created by Doig. The defendants challenged plaintiff’s expert witness as not reliable or relevant under Federal Rule of Evidence 702. Fletcher v. Doig, 13 C 3270, 2016 U.S. Dist. LEXIS 95081 (N.D. Ill. July 21, 2016).

Peter Bartlow, the plaintiff’s expert witness on authenticity, was short on academic credentials. He had gone to college, and finished only one year of graduate study in art history. Bartlow did, however, have 40 years in experience in appraisal and authentication. Fletcher, at *3-4. Beyond qualifications, the defendants complained that Bartlow’s method was

(1) invented for the case,

(2) was too “generic” to establish authenticity, and

(3) failed to show that any claimed generic feature was unique to the work of the artist in question, Peter M. Doig.

The trial court rebuffed this challenge by noting that Peter Bartlow did not have to be an expert specifically in Doig’s work. Fletcher at *7. Similarly, the trial court rejected the defendants’ suggestion that the disputed work must exhibit “unique” features of Doig’s ouevre. Bartlow had made a legally sufficient case for his opinions based upon a qualitative analysis of 45 acknowledged works, using specific qualitative features of 11 known works. Id. At *10. Specifically, Bartlow compared types of paint, similarities in styles, shapes and positioning, and “repeated lineatures” by superimposing lines from known paintings to the questioned ones. Id. With respect to the last of these approaches, the trial court found that Bartlow’s explanation that the approach of superimposing lines to show similarity was simply a refinement of methods commonly used by art appraisers.

By comparison with Thomas Hoving’s subjective auditory methodology, as explained in Who the Fuck, Bartlow’s approach was positively brilliant, even if the challenged methodologies left much to be desired. For instance, Bartlow compared one disputed painting with 45 or so paintings of accepted provenance. No one tested Bartlow’s ability, blinded to provenance, to identify true and false positives of Doig paintings. See “The Eleventh Circuit Confuses Adversarial and Methodological Bias, Manifestly Erroneously” (June 6, 2015); see generally Christopher Robertson & Aaron Kesselheim, Blinding as a Solution to Bias: Strengthening Biomedical Science, Forensic Science, and Law (2016).

Interestingly, the Rule 702 challenges in Fletcher were in a case slated to be tried by the bench. The trial court thus toasted the chestnut that trial courts have even greater latitude in admitting expert witness opinion testimony in bench trials, in which “the usual concerns of [Rule 702] – keeping unreliable testimony from the jury – are not present.” Fletcher at *3 (citing Metavante Corp. v. Emigrants Savings Bank, 619 F.3d 648, 670 (7th Cir. 2010)). Citing Seventh Circuit precedent, the trial court, in Fletcher, asserted that the need to rule on admissibility before trial was lessened in a bench trial. Id. (citing In re Salem, 465 F.3d 767, 777 (7th Cir. 2006)). The courts that have taken this position have generally failed to explain why the standard for granting or denying a Rule 702 challenge should be different in a bench trial. Clearly, a bench trial can be just as much a waste of time, money, and energy as a jury trial. Even more clearly, judges can be, and are, snookered by misleading expert witness opinions, and they are also susceptible to their own cognitive biases and the false allure of unreliable opinion testimony, built upon invalid inferences. Men and women do not necessarily see more clearly when wearing black robes, but they can achieve some measure of objectivity by explaining and justifying their gatekeeping opinions in writing, subject to public review, comment, and criticism.

[1] See, e.g. Lees v. Carthage College, 714 F.3d 516, 525 (7th Cir. 2013) (holding that an expert witness’s testimony on premises security involved non-scientific expertise and knowledge that did “not easily admit of rigorous testing and replication”).

[2] See, e.g., National Academies of Science, Strengthening Forensic Science in the United States: A Path Forward (2009).

Posted in Cognitive Biases, Expert Witnesses, Historian Testimony, Rule 702 | Comments Off on Art Historian Expert Testimony

The LoGiudice Inquisitiorial Subpoena & Its Antecedents in N.Y. Law

July 14th, 2016

The plaintiffs’ bar’s inquisition into funding has been a recurring theme in the asbestos and other litigations.[1] It is thus interesting to compare the friendly reception Justice Moulton gave plaintiffs’ subpoena in LoGiudice[2] with the New York courts’ relatively recent hostility toward a defendant’s subpoena to Mt. Sinai School of Medicine.

A few years ago, Justice Sherry Heitler quashed a defendant’s attempt to subpoena information from the archives of a deceased, former faculty member of the Mount Sinai School of Medicine (“Mt. Sinai”), in Reyniak v. Barnstead Internat’l, No. 102688-08, 2010 NY Slip Op 50689, 2010 WL 1568424 (N.Y. Sup. Ct. Apr. 6, 2010). In a cursory opinion, Justice Heitler cited institutional expense, chilling of research, and scholars’ fears that their unpublished notes, ideas, and observations would become public as a result of litigation. Heitler relied upon and followed an earlier New York state court’s decision that adopted a rather lopsided “balancing” analysis, which permitted the New York courts to ignore the legitimate needs of defendants for access to underlying data.[3]

Remarkably, Justice Heitler failed to cite a federal appellate court’s subsequent decision, which upheld the tobacco companies’ subpoena to Mount Sinai.[4] Her opinion also ignored the important context of the asbestos litigation, in which Selikoff, long since deceased, played a crucial role in fomenting and perpetuating litigation, with tendentious publications and pronouncements. Some might say, “manufacturing certainty.” Perpetuating the Litigation Industry’s Selikoff mythology, Justice Heitler described Selikoff as a ground breaking asbestos researcher, but she either ignored, or was ignorant of, his testimonial adventures, his attempts to influence litigation with ex parte meetings with presiding judges, and his other questionable litigation-related conduct.

Selikoff’s participation in litigation was not always above board.  His supposedly ground-breaking work was funded by the insulator’s union, which also sought him out as a testifying expert witness. Among his many testimonial adventures,[5] Selikoff testified as early as 1966 that asbestos causes colorectal cancer, and that it caused a specific claimant’s colorectal cancer. See “Health Hazard Progress Notes: Compensation Advance Made in New York State,” 16(5) Asbestos Worker 13 (May 1966) (thanking Selikoff for his having given testimony to support an insulator’s claim that asbestos caused his colorectal cancer). To be sure, Selikoff made his litigation claims in the scientific literature as well, but without any acknowledgement of his involving in litigation involving this very issue, and his funding by the asbestos union.[6]

Given the dubious provenance of many of Selikoff’s opinions,[7] the disparate treatment of the subpoenas in LoGuidice and Reyniak is irreconcilable. The inflated prestige of Selikoff and Mount Sinai blinded the New York state trial courts to Selikoff’s role in litigation and his biased assessments in science. The judicial hypocrisy may well be the consequence of how the academic community has promoted Selikoff’s reputation, while working assiduously to undermine the reputations of anyone who has been connected with the defense of occupational disease claims. Consider, for instance, how Labor (Marxist) historians have railed against the role that Dr. Anthony Lanza played in personal injury litigation following the Gauley Bridge tunnel construction.  See Jock McCulloch and Geoffrey Tweedale, “Anthony J. Lanza, Silicosis and the Gauley Bridge ‘Nine’,” 27 Social History of Medicine 86 (2013). While these historians deplore Lanza, however, they laud Selikoff. See “British Labor Historians Belaboring American Labor History – Gauley Bridge” (Oct. 14, 2013). Politics and occupational disease litigation are like that.

[1] See In re All Litigation filed by Maune, Raichle, Hartley, French & Mudd LLC v. 3M Co., No. 5-15-0235, Ill. App., 5th Dist.; 2016 Ill. App. Unpub. LEXIS 1392 (June 30, 2016); “Engineers for Automakers Must Unredact Agendas in Madison County Asbestos Litigation,” Madison County Record (July 2016); Lynn A. Lenhart, “Meeting Agendas Between Non-Party Consultant and Counsel for Asbestos Friction Clients Not Privileged” (July 5, 2016).  See also Weitz & Luxenberg P.C. v. Georgia-Pacific LLC, 2013 WL 2435565, 2013 NY Slip Op 04127 (June 6, 2013), aff’d, 2013 WL 2435565 (N.Y. App. Div., 1st Dep’t June 6, 2013); “A Cautionary Tale on How Not to Sponsor a Scientific Study for Litigation” (June 21, 2013).

[2] LoGiudice v. American Talc Co., No. 190253/2014, 2016 N.Y. Misc. LEXIS 2360, (N.Y. Sup., N.Y. Cty., June 20, 2016).

[3] See In re R.J. Reynolds Tobacco Co., 136 Misc 2d 282, 285, 518 N.Y.S.2d 729 (Sup. Ct., N.Y. Cty. 1987); see also In re New York County Data Entry Worker Prod. Liab.Litig., No. 14003/92, 1994 WL 87529 (N.Y. Sup. Ct. N.Y. Cty. Jan 31, 1994) (denying discovery because “special circumstances,” vaguely defined were absent).

[4] Mount Sinai School of Medicine v. The American Tobacco Co., 866 F.2d 552 (2d Cir. 1889).

[5] “Selikoff and the Mystery of the Disappearing Testimony” (Dec. 3, 2010).

[6] See, e.g., Irving J. Selikoff, “Epidemiology of gastrointestinal cancer,” 9 Envt’l Health Persp. 299 (1974) (arguing for his causal conclusion between asbestos and all gastrointestinal cancers).movie Her trailer

[7] See generally “Scientific Prestige, Reputation, Authority & The Creation of Scientific Dogmas” (Oct. 4, 2014); “Historians Should Verify Not Vilify or Abilify – The Difficult Case of Irving Selikoff” (Jan. 4, 2014).

Posted in Asbestos, Cognitive Biases, Conflicts of Interest, Expert Witness Discovery | Comments Off on The LoGiudice Inquisitiorial Subpoena & Its Antecedents in N.Y. Law

Mens Rea Defense – Good Heart (?) and Empty Head

July 11th, 2016

Extreme Carelessness Versus Gross Negligence

The case of Hilary Clinton offers an interesting fact set for exploring jurisprudential questions about the differences among intentional, reckless, and negligent misconduct. Of course, Clinton’s malfeasance, regardless of the attributed mens rea, has received a good deal of publicity, and ultimately, there should be an exemplary factual record, which can be used to explore the different culpable states of mind.

In her column yesterday, Maureen Dowd captured how many rational United States voters must feel about the irrationality of our national politics, and the Clinton email scandal.[1]  James Comey, Director of the FBI, detailed the facts that would probably require President Obama to fire Hilary Clinton if she were still Secretary of State in his administration.  Instead, the President is endorsing Clinton to be his successor. Of course, Obama could put the nation at ease, however, by revealing that he never allowed Clinton to have access to REALLY confidential information because he just did not trust her. Perhaps that might improve public perception of his judgment while exculpating her.

Dowd was commenting upon Comey’s conclusion that Hillary Clinton’s use of email for State Department confidential and classified communications, over her own, private e-mail server to handle work-related communications was “extremely careless.” See James B. Comey, “Statement by FBI Director on the Investigation of Secretary Hillary Clinton’s Use of a Personal E-Mail System,” FBI Press Office, Washington, D.C. (July 05, 2016). In his press conference, however, Comey announced that the FBI would not recommend criminal charges because in his view Ms. Clinton’s conduct was not proscribed by pertinent statutes. And of course, Obama’s Attorney General, Loretta Lynch, fresh from a tête-à-tête with Ms. Clinton’s husband, who had nominated her to the position of United States Attorney back in 1999, agreed with Comey in a New York minute.

Comey framed his political indictment of extreme carelessness in a way to suggest that although Clinton might not be worthy of a security clearance, she should not be prosecuted:

“Although we did not find clear evidence that Secretary Clinton or her colleagues intended to violate laws governing the handling of classified information, there is evidence that they were extremely careless in their handling of very sensitive, highly classified information.”

Comey acknowledged that “there is evidence of potential violations of the statutes regarding the handling of classified information,” but offered his judgment “that no reasonable prosecutor would bring such a case.”[2]

But why is it that no reasonable prosecute would indict? This is where the Clinton scandal raises questions about inferring mental states and becomes jurisprudentially interesting.

Comey was clear that Clinton had put confidential, classified information at risk, by using her email on foreign soil, where sophisticated, “hostile actors” could have hacked her account without her being able to certify otherwise. Cybersecurity experts were less “politic,” and more willing to go beyond “possible” to claim “probable” hacking had taken place. Clinton had no full-time cyber-security professional monitoring her email system.[3]

Comey ended his performance by claiming that only the facts mattered; opinions were irrelevant. “Just the facts, ma’am.” But his judgments about reasonable prosecutors, obstruction, and disloyalty are not facts; they are opinions, and particularly, they are opinions that involve inferences about mental states, mens rea, and motives.

Comey offered precedent, in the form of prior prosecutions, which according to Comey, all involved elements of intentional conduct, including willful mishandling of classified information, indications of disloyalty to the United States, or efforts to obstruct justice. And Comey just did not see those elements in the Hilary Clinton case.

Were the inferences to these putatively missing elements truly unwarranted? Was Comey really looking or was this a case of cognitive bias or willful blindness? After all, Comey was compelled “on the facts,” to acknowledge that there were over 100 Clinton emails with classified information turned over to the State Department, and a few additional emails found with classified information, which were not turned over to State.  Of course no one could say what was in the emails that had been spoliated by Clinton or under her supervision.

Clinton clearly had attempted to obstruct the investigation into her unlawful conduct. She repeatedly lied about her motive for eschewing State Department regulations and protocols. She destroyed evidence.  She lied about the content of the emails and their security status. Clinton’s disloyalty to the country was manifest. She adopted a private, unsecured email system not only for her own convenience, for the needs of her own future political candidacy, and so that she could provide access to confidents not in the government, such as unapproved actors, Sidney Blumenthal and William J. Clinton.

Given that the email server belonged to the Clintons, not the State Department, former President Bill Clinton, could check into his Chappaqua, New York, system to catch up on the latest diplomatic initiatives before he set off to give a six-figure speech to foreign potentates. Bill Clinton claimed he does not send emails, but that does not mean he does not read emails, on which may have been blind copied. How convenient to bcc Bill on emails sent to Barack, to get keep him “in the loop”?

Comey’s suggestion that Clinton did not meet the mens rea requirement of the pertinent criminal statute would not seem to hold up under scrutiny. The provisions of the federal criminal code define punishable conduct for “gathering, transmitting, or losing defense information.” 18 U.S. Code § 793. This provision does require intent or gross negligence, which Comey seemed to suggest were absent or not readily proven in the Clinton case. And yet, Clinton intended to gather and transmit defense information contrary to law.  Actually “losing information” is not a required element, although even there, Clinton lost information that sat on her server when she chose to destroy it rather than maintain it, as was legally required.

As for gross negligence, Comey himself made the case, although he characterized Clinton’s conduct as “extremely careless.” There does not seem to be any meaningful distinction between gross negligence and extreme carelessness. The current Wikipedia entry for gross negligence illustrates the subtle, sometimes evanescent distinctions among “negligence,” “gross negligence,” and “recklessness”:

 “Gross negligence is legally culpable carelessness that shows a conscious and voluntary disregard of the need to use reasonable care, and likely to cause foreseeable grave injury or harm. The difference between “negligence” and “gross negligence” may be subjective since it is a matter of degree. Negligence is the opposite of diligence, or being careful. The standard of ordinary negligence is what conduct deviates from the proverbial “reasonable person.” By analogy, if somebody has been grossly negligent, that means they have fallen so far below the ordinary standard of care that one can expect, to warrant the label of being “gross.” Prosser and Keeton describe gross negligence as being “the want of even slight or scant care”, and note it as having been described as a lack of care that even a careless person would use. They further note that while some jurisdictions equate the culpability of gross negligence with that of recklessness, most simply differentiate it from simple negligence in its degree.[1]”

Jurisprudes struggle to define gross negligence and distinguish it from ordinarily negligence and recklessness. Consensus and precise definitions are hard to find, and some of the case law definitions appear vacuous or circular. In many analyses, the grossness of someone’s negligence may be something recognized when encountered, much like Potter Stewart’s obscenities. See Jacobellis v. Ohio, 378 U.S. 184, 197 (1964).

The recent litigation over the 2010 oil spill in the Gulf of Mexico, caused by the oil rig, Deepwater Horizon, illustrates the struggle to define gross negligence:

“Gross negligence is a nebulous term that is defined in a multitude of ways, depending on the legal context and the jurisdiction. However, when the “cluster of ideas” surrounding “gross negligence” is considered, the prevailing notion is that gross negligence differs from ordinary negligence in terms of degree, and both are different in kind from reckless, wanton, and willful misconduct.

[…]

Gross negligence, like ordinary negligence, requires only objective, not subjective, proof. While ordinary negligence is a failure to exercise the degree of care that someone of ordinary prudence would have exercised in the same circumstances, gross negligence is an extreme departure from the care required under the circumstances or a failure to exercise even slight care. Thus, the United States contends that gross negligence differs from ordinary negligence only in degree, not in kind.”

In re Oil Spill by Oil Rig Deepwater Horizon in Gulf of Mexico, 2014 WL 4375933 (E.D. La. Sept. 4, 2014).

Some commentators have argued that reckless and careless are synonyms.[4] This argument ignores important distinctions drawn in the criminal law, and in the law of torts. Perhaps the clearest distinction between recklessness and negligence is set out in the Model Penal Code’s definitions of the kinds of culpability.  MPC  § 2.02 (2). Subsection (c) defines “recklessly” as follows:

“A person acts recklessly with respect to a material element of an offense when he consciously disregards a substantial and unjustifiable risk that the material element exists or will result from his conduct. The risk must be of such a nature and degree that, considering the nature and purpose of the actor’s conduct and the circumstances known to him, its disregard involves a gross deviation from the standard of conduct that a law-abiding person would observe in the actor’s situation.”

And Subsection (d) defines “negligently” as follows:

“A person acts negligently with respect to a material element of an offense when he should be aware of a substantial and unjustifiable risk that the material element exists or will result from his conduct. The risk must be of such a nature and degree that the actor’s failure to perceive it, considering the nature and purpose of his conduct and the circumstances known to him, involves a gross deviation from the standard of care that a reasonable person would observe in the actor’s situation.”

So both recklessness and (criminal) negligence involve “gross” deviations, but recklessness requires a greater subjective awareness, whereas negligence can be shown using an objective, “reasonable person” test.

Given Comey’s recitation of “just the facts,” and adding in his Capitol Hill testimony, it would seem that a rookie prosecutor could indict and convict Ms. Clinton of intentional or grossly negligent mishandling of confidential information, as well as a conspiracy to cover it up.

Of course, there might be good political reasons not to indict Clinton, including that it might cause the election of Donald Trump, but that would require adverting to facts beyond the Clinton case.

[1] Maureen Dowd, “The Clinton Contamination,” New York Times (July 10, 2016).

[2] See Daniel Fisher, “FBI Calls Hillary’s E-Mail Habits ‘Extremely Careless’ But Not Criminal,” Forbes (July 5, 2016).

[3] David E. Sanger, “Hillary Clinton’s Email Was Probably Hacked, Experts Say,” New York Times (July 6, 2016).

[4] See Craig Bannister, “A President Can’t Be Reckless” National Review (July 2016).

Posted in Cognitive Biases | Comments Off on Mens Rea Defense – Good Heart (?) and Empty Head

The Eleventh Circuit Confuses Adversarial and Methodological Bias, Manifestly Erroneously

June 6th, 2015

The Eleventh Circuit’s decision in Adams v. Laboratory Corporation of America, is disturbing on many levels. Adams v. Lab. Corp. of Am., 760 F.3d 1322 (11th Cir. 2014). Professor David Bernstein has already taken the Circuit judges to task for their failure to heed the statutory requirements of Federal Rule of Evidence 702. See David Bernstein, “A regrettable Eleventh Circuit expert testimony ruling, Adams v. Lab. Corp. of America,” Wash. Post (May 26, 2015). Sadly, the courts’ strident refusal to acknowledge the statutory nature of Rule 702, and the Congressional ratification of the 2000 amendments to Rule 702, have become commonplace in the federal courts. Ironically, the holding of the Supreme Court’s decision in Daubert itself was that the lower courts were not free to follow common law that had not been incorporated into the first version of the Rule 702.

There is much more wrong with the Adams case than just a recalcitrant disregard for the law: the Circuit displayed an equally distressing disregard for science. The case started as a negligent failure to diagnose cervical cancer claim against defendant Laboratory Corporation of America. Plaintiffs claimed that the failure to diagnose cancer led to delays in treatment, which eroded Mrs. Adam’s chance for a cure.

Before the Adams case arose, two professional organizations, the College of American Pathologists (CAP) and the American Society of Cytopathology (ASC) issued guidelines about how an appropriate retrospective review should be conducted. Both organizations were motivated by two concerns: protecting their members from exaggerated, over-extended, and bogus litigation claims, as well as by a scientific understanding that a false-negative finding by a cytopathologist does not necessarily reflect a negligent interpretation of a Pap smear[1]. Both organizations called for a standard of blinded review in litigation to protect against hind-sight bias. The Adams retained a highly qualified pathologist, Dr. Dorothy Rosenthal, who with full knowledge of the later diagnosis and the professional guidelines, reviewed the earlier Pap smears that were allegedly misdiagnosed as non-malignant. 760 F.3d at 1326. Rosenthal’s approach violated the CAP and ASC guidelines, as well as common sense.

The district judge ruled that Rosenthal’s approach was little more than an ipse dixit, and a subjective method that could not be reviewed objectively. Adams v. Lab. Corp. of Am., No. 1:10-CV-3309-WSD, 2012 WL 370262, at *15 (N.D. Ga. Feb. 3, 2012). In a published per curiam opinion, the Eleventh Circuit reversed, holding that the district judge’s analysis of Rosenthal’s opinion was “manifestly erroneous.” 760 F.3d at 1328. Judge Garza, of the Fifth Circuit, sitting by designation, concurred to emphasize his opinion that Rosenthal did not need a methodology, as long as she showed up with her qualifications and experience to review the contested Pap smears.

The Circuit opinion is a model of conceptual confusion. The judges refer to the professional society guidelines, but never provide citations. (See note 1, infra.). The Circuit judges are obviously concerned that the professional societies are promulgating standards to be used in judging claims against their own members for negligent false-negative interpretations of cytology or pathology. What the appellate judges failed to recognize, however, is that the professional societies had a strong methodological basis for insisting upon “blinded” review of the slides in controversy. Knowledge of the outcome must of necessity bias any subsequent review, such as plaintiffs’ expert witness, Rosenthal. Even a cursory reading of the two guidelines would have made clear that they had been based on more than simply a desire to protect members; they were designed to protect members against bogus claims, and cited data in support of their position[2]. Subsequent to the guidelines, several publications have corroborated the evidence-based need for blinded review[3].

The concepts of sensitivity, specificity, and positive predictive value are inherent in any screening procedure; they are very much part of the methodology of screening. These measures, along with statistical analyses of concordance and discordance among experienced cytopathologists, can be measured and assessed for accuracy and reliability. The Circuit judges in Adams, however, were blinded (in a bad way) to the scientific scruples that govern screenings. The per curiam opinion suggests that:

“[t]he only arguably appreciable differences between Dr. Rosenthal’s method and the review method for LabCorp’s cytotechnologists is that Dr. Rosenthal (1) already knew that the patient whose slides she was reviewing had developed cancer and (2) reviewed slides from just one patient. Those differences relate to the lack of blinded review, which we address later.”

760 F.3d at 1329 n. 10. And when the judges addressed the lack of blinded review, they treated hindsight bias, a cognitive bias and methodological flaw in the same way as they would have trial courts and litigants treat Dr. Rosenthal’s “philosophical bent” in favor of cancer patients — as “a credibility issue for the jury.” Id. at 1326-27, 1332. This conflation of methodological bias with adversarial bias, however, is a prescription for eviscerating judicial gatekeeping of flawed opinion testimony. Judge Garza, in a concurring opinion, would have gone further and declared that plaintiffs’ expert witness Rosenthal had no methodology and thus she was free to opine ad libitum.

Although Rosenthal’s “philosophical bent” might perhaps be left to the crucible of cross-examination, hindsight review bias could and should have been eliminated by insisting that Rosenthal wear the same “veil of ignorance” of Mrs. Adam’s future clinical course, which the defendant wore when historically evaluating the plaintiff’s Pap smears. Here Rosenthal’s adversarial bias was very probably exacerbated by her hindsight bias, and the Circuit missed a valuable opportunity to rein in both kinds of bias.

Certainly in other areas of medicine, such as radiology, physicians are blinded to the correct interpretation and evaluated on their ability to line up with a gold standard. The NIOSH B-reader examination, for all its problems, at least tries to qualify physicians in the use of the International Labor Organization’s pneumoconiosis scales for interpreting plain-film radiographs for pulmonary dust diseases, by having them read and interpret films blinded to the NIOSH/ILO consensus interpretation.

[1] See Patrick L. Fitzgibbons & R. Marshall Austin, “Expert review of histologic slides and Papanicolaou tests in the context of litigation or potential litigation — Surgical Pathology Committee and Cytopathology Committee of the College of American Pathologists,” 124 Arch. Pathol. Lab. Med. 1717 (2000); American Society of Cytopathology, “Guidelines for Review of Gyn Cytology Samples in the Context of Litigation or Potential Litigation” (2000).

[2] The CAP guideline, for instance, cited R. Marshall Austin, “Results of blinded rescreening of Papanicolaou smears versus biased retrospective review,” 121 Arch. Pathol. Lab. Med. 311 (1997).

[3] Andrew A. Renshaw, K.M Lezon, and D.C. Wilbur, “The human false-negative rate of rescreening Pap tests: Measured in a two-arm prospective clinical trial,” 93 Cancer (Cancer Cytopathol.) 106 (2001); Andrew A. Renshaw, Mary L. Young, and E. Blair Holladay, “Blinded review of Papanicolaou smears in the context of litigation: Using statistical analysis to define appropriate thresholds,” 102 Cancer Cytopathology 136 (2004) (showing that data from blinded reviews can be interpreted in a statistically appropriate way, and defining standards to improve the accuracy and utility of blinded reviews); D. V. Coleman & J. J. R. Poznansky, “Review of cervical smears from 76 women with invasive cervical cancer: cytological findings and medicolegal implications,” 17 Cytopathology 127 (2006); Andrew A. Renshaw, “Comparing Methods to Measure Error in Gynecologic Cytology and Surgical Pathology,” 130 Arch. Path. & Lab. Med. 626 (2009).

Posted in Cognitive Biases, Rule 702 | Comments Off on The Eleventh Circuit Confuses Adversarial and Methodological Bias, Manifestly Erroneously

Judicial Control of the Rate of Error in Expert Witness Testimony

May 28th, 2015

In Daubert, the Supreme Court set out several criteria or factors for evaluating the “reliability” of expert witness opinion testimony. The third factor in the Court’s enumeration was whether the trial court had considered “the known or potential rate of error” in assessing the scientific reliability of the proffered expert witness’s opinion. Daubert v. Merrell Dow Pharmaceuticals, Inc., 509 U.S. 579, 593 (1993). The Court, speaking through Justice Blackmun, failed to provide much guidance on the nature of the errors subject to gatekeeping, on how to quantify the errors, and on to know how much error was too much. Rather than provide a taxonomy of error, the Court lumped “accuracy, validity, and reliability” together with a grand pronouncement that these measures were distinguished by no more than a “hen’s kick.” Id. at 590 n.9 (1993) (citing and quoting James E. Starrs, “Frye v. United States Restructured and Revitalized: A Proposal to Amend Federal Evidence Rule 702,” 26 Jurimetrics J. 249, 256 (1986)).

The Supreme Court’s failure to elucidate its “rate of error” factor has caused a great deal of mischief in the lower courts. In practice, trial courts have rejected engineering opinions on stated grounds of their lacking an error rate as a way of noting that the opinions were bereft of experimental and empirical evidential support[1]. For polygraph evidence, courts have used the error rate factor to obscure their policy prejudices against polygraphs, and to exclude test data even when the error rate is known, and rather low compared to what passes for expert witness opinion testimony in many other fields[2]. In the context of forensic evidence, the courts have rebuffed objections to random-match probabilities that would require that such probabilities be modified by the probability of laboratory or other error[3].

When it comes to epidemiologic and other studies that require statistical analyses, lawyers on both sides of the “v” frequently misunderstand p-values or confidence intervals to provide complete measures of error, and ignore the larger errors that result from bias, confounding, study validity (internal and external), inappropriate data synthesis, and the like[4]. Not surprisingly, parties fallaciously argue that the Daubert criterion of “rate of error” is satisfied by expert witness’s reliance upon studies that in turn use conventional 95% confidence intervals and measures of statistical significance in p-values below 0.05[5].

The lawyers who embrace confidence intervals and p-values as their sole measure of error rate fail to recognize that confidence intervals and p-values are means of assessing only one kind of error: random sampling error. Given the carelessness of the Supreme Court’s use of technical terms in Daubert, and its failure to engage in the actual evidence at issue in the case, it is difficult to know whether the Court intended to suggest that random error was the error rate it had in mind[6]. The statistics chapter in the Reference Manual on Scientific Evidence helpfully points out that the inferences that can be drawn from data turn on p-values and confidence intervals, as well as on study design, data quality, and the presence or absence of systematic errors, such as bias or confounding.  Reference Manual on Scientific Evidence at 240 (3d 2011) [Manual]. Random errors are reflected in the size of p-values or the width of confidence intervals, but these measures of random sampling error ignore systematic errors such as confounding and study biases. Id. at 249 & n.96.

The Manual’s chapter on epidemiology takes an even stronger stance: the p-value for a given study does not provide a rate of error or even a probability of error for an epidemiologic study:

“Epidemiology, however, unlike some other methodologies—fingerprint identification, for example—does not permit an assessment of its accuracy by testing with a known reference standard. A p-value provides information only about the plausibility of random error given the study result, but the true relationship between agent and outcome remains unknown. Moreover, a p-value provides no information about whether other sources of error – bias and confounding – exist and, if so, their magnitude. In short, for epidemiology, there is no way to determine a rate of error.”

Manual at 575. This stance seems not entirely justified given that there are Bayesian approaches that would produce credibility intervals accounting for sampling and systematic biases. To be sure, such approaches have their own problems and they have received little to no attention in courtroom proceedings to date.

The authors of the Manual’s epidemiology chapter, who are usually forgiving of judicial error in interpreting epidemiologic studies, point to one United States Court of Appeals case that fallaciously interpreted confidence intervals magically to quantify bias and confounding in a Bendectin birth defects case. Id. at 575 n. 96[7]. The Manual could have gone further to point out that, in the context of multiple studies, of different designs and analyses, cognitive biases involved in evaluating, assessing, and synthesizing the studies are also ignored by statistical measures such as p-values and confidence intervals. Although the Manual notes that assessing the role of chance in producing a particular set of sample data is “often viewed as essential when making inferences from data,” the Manual never suggests that random sampling error is the only kind of error that must be assessed when interpreting data. The Daubert criterion would appear to encompass all varieties or error, not just random error.

The Manual’s suggestion that epidemiology does not permit an assessment of the accuracy of epidemiologic findings misrepresents the capabilities of modern epidemiologic methods. Courts can, and do, invoke gatekeeping approaches to weed out confounded study findings. See “Sorting Out Confounded Research – Required by Rule 702” (June 10, 2012). The “reverse Cornfield inequality” was an important analysis that helped establish the causal connection between tobacco smoke and lung cancer[8]. Olav Axelson studied and quantified the role of smoking as a confounder in epidemiologic analyses of other putative lung carcinogens.[9] Quantitative methods for identifying confounders have been widely deployed[10].

A recent study in birth defects epidemiology demonstrates the power of sibling cohorts in addressing the problem of residual confounding from observational population studies with limited information about confounding variables. Researchers looking at various birth defect outcomes among offspring of women who used certain antidepressants in early pregnancy generally found no associations in pooled data from Iceland, Norway, Sweden, Finland, and Denmark. A putative association between maternal antidepressant use and a specific kind of cardiac defect (right ventricular outflow tract obstruction or RVOTO) did appear in the overall analysis, but was reversed when the analysis was limited to the sibling subcohort. The study found an apparent association between RVOTO defects and first trimester maternal exposure to selective serotonin reuptake inhibitors, with an adjusted odds ratio of 1.48 (95% C.I., 1.15, 1.89). In the adjusted analysis for siblings, the study found an OR of 0.56 (95% C.I., 0.21, 1.49) in an adjusted sibling analysis[11]. This study and many others show how creative analyses can elucidate and quantify the direction and magnitude of confounding effects in observational epidemiology.

Systematic bias has also begun to succumb to more quantitative approaches. A recent guidance paper by well-known authors encourages the use of quantitative bias analysis to provide estimates of uncertainty due to systematic errors[12].

Although the courts have failed to articulate the nature and consequences of erroneous inference, some authors would reduce all of Rule 702 (and perhaps 704, 403 as well) to a requirement that proffered expert witnesses “account” for the known and potential errors in their opinions:

“If an expert can account for the measurement error, the random error, and the systematic error in his evidence, then he ought to be permitted to testify. On the other hand, if he should fail to account for any one or more of these three types of error, then his testimony ought not be admitted.”

Mark Haug & Emily Baird, “Finding the Error in Daubert,” 62 Hastings L.J. 737, 739 (2011).

Like most antic proposals to revise Rule 702, this reform vision shuts out the full range of Rule 702’s remedial scope. Scientists certainly try to identify potential sources of error, but they are not necessarily very good at it. See Richard Horton, “Offline: What is medicine’s 5 sigma?” 385 Lancet 1380 (2015) (“much of the scientific literature, perhaps half, may simply be untrue”). And as Holmes pointed out[13], certitude is not certainty, and expert witnesses are not likely to be good judges of their own inferential errors[14]. Courts continue to say and do wildly inconsistent things in the course of gatekeeping. Compare In re Zoloft (Setraline Hydrochloride) Products, 26 F. Supp. 3d 449, 452 (E.D. Pa. 2014) (excluding expert witness) (“The experts must use good grounds to reach their conclusions, but not necessarily the best grounds or unflawed methods.”), with Gutierrez v. Johnson & Johnson, 2006 WL 3246605, at *2 (D.N.J. November 6, 2006) (denying motions to exclude expert witnesses) (“The Daubert inquiry was designed to shield the fact finder from flawed evidence.”).

[1] See, e.g., Rabozzi v. Bombardier, Inc., No. 5:03-CV-1397 (NAM/DEP), 2007 U.S. Dist. LEXIS 21724, at *7, *8, *20 (N.D.N.Y. Mar. 27, 2007) (excluding testimony from civil engineer about boat design, in part because witness failed to provide rate of error); Sorto-Romero v. Delta Int’l Mach. Corp., No. 05-CV-5172 (SJF) (AKT), 2007 U.S. Dist. LEXIS 71588, at *22–23 (E.D.N.Y. Sept. 24, 2007) (excluding engineering opinion that defective wood-carving tool caused injury because of lack of error rate); Phillips v. Raymond Corp., 364 F. Supp. 2d 730, 732–33 (N.D. Ill. 2005) (excluding biomechanics expert witness who had not reliably tested his claims in a way to produce an accurate rate of error); Roane v. Greenwich Swim Comm., 330 F. Supp. 2d 306, 309, 319 (S.D.N.Y. 2004) (excluding mechanical engineer, in part because witness failed to provide rate of error); Nook v. Long Island R.R., 190 F. Supp. 2d 639, 641–42 (S.D.N.Y. 2002) (excluding industrial hygienist’s opinion in part because witness was unable to provide a known rate of error).

[2] See, e.g., United States v. Microtek Int’l Dev. Sys. Div., Inc., No. 99-298-KI, 2000 U.S. Dist. LEXIS 2771, at *2, *10–13, *15 (D. Or. Mar. 10, 2000) (excluding polygraph data based upon showing that claimed error rate came from highly controlled situations, and that “real world” situations led to much higher error (10%) false positive error rates); Meyers v. Arcudi, 947 F. Supp. 581 (D. Conn. 1996) (excluding polygraph in civil action).

[3] See, e.g., United States v. Ewell, 252 F. Supp. 2d 104, 113–14 (D.N.J. 2003) (rejecting defendant’s objection to government’s failure to quantify laboratory error rate); United States v. Shea, 957 F. Supp. 331, 334–45 (D.N.H. 1997) (rejecting objection to government witness’s providing separate match and error probability rates).

[4] For a typical judicial misstatement, see In re Zoloft Products, 26 F. Supp.3d 449, 454 (E.D. Pa. 2014) (“A 95% confidence interval means that there is a 95% chance that the ‘‘true’’ ratio value falls within the confidence interval range.”).

[5] From my experience, this fallacious argument is advanced by both plaintiffs’ and defendants’ counsel and expert witnesses. See also Mark Haug & Emily Baird, “Finding the Error in Daubert,” 62 Hastings L.J. 737, 751 & n.72 (2011).

[6] See David L. Faigman, et al. eds., Modern Scientific Evidence: The Law and Science of Expert Testimony § 6:36, at 359 (2007–08) (“it is easy to mistake the p-value for the probability that there is no difference”)

[7] Brock v. Merrell Dow Pharmaceuticals, Inc., 874 F.2d 307, 311-12 (5th Cir. 1989), modified, 884 F.2d 166 (5th Cir. 1989), cert. denied, 494 U.S. 1046 (1990). As with any error of this sort, there is always the question whether the judges were entrapped by the parties or their expert witnesses, or whether the judges came up with the fallacy on their own.

[8] See Joel B Greenhouse, “Commentary: Cornfield, Epidemiology and Causality,” 38 Internat’l J. Epidem. 1199 (2009).

[9] Olav Axelson & Kyle Steenland, “Indirect methods of assessing the effects of tobacco use in occupational studies,” 13 Am. J. Indus. Med. 105 (1988); Olav Axelson, “Confounding from smoking in occupational epidemiology,” 46 Brit. J. Indus. Med. 505 (1989); Olav Axelson, “Aspects on confounding in occupational health epidemiology,” 4 Scand. J. Work Envt’l Health 85 (1978).

[10] See, e.g., David Kriebel, Ariana Zeka1, Ellen A Eisen, and David H. Wegman, “Quantitative evaluation of the effects of uncontrolled confounding by alcohol and tobacco in occupational cancer studies,” 33 Internat’l J. Epidem. 1040 (2004).

[11] Kari Furu, Helle Kieler, Bengt Haglund, Anders Engeland, Randi Selmer, Olof Stephansson, Unnur Anna Valdimarsdottir, Helga Zoega, Miia Artama, Mika Gissler, Heli Malm, and Mette Nørgaard, “Selective serotonin reuptake inhibitors and ventafaxine in early pregnancy and risk of birth defects: population based cohort study and sibling design,” 350 Brit. Med. J. 1798 (2015).

[12] Timothy L.. Lash, Matthew P. Fox, Richard F. MacLehose, George Maldonado, Lawrence C. McCandless, and Sander Greenland, “Good practices for quantitative bias analysis,” 43 Internat’l J. Epidem. 1969 (2014).

[13] Oliver Wendell Holmes, Jr., Collected Legal Papers at 311 (1920) (“Certitude is not the test of certainty. We have been cock-sure of many things that were not so.”).

[14] See, e.g., Daniel Kahneman & Amos Tversky, “Judgment under Uncertainty:  Heuristics and Biases,” 185 Science 1124 (1974).

Posted in Cognitive Biases, Expert Witnesses, Reference Manual on Scientific Evidence, Rule 702, Scientific Evidence | Comments Off on Judicial Control of the Rate of Error in Expert Witness Testimony

Science as Adversarial Process versus Group Think

May 7th, 2015

Climate scientists, at least those scientists who believe that climate change is both real and an existential threat to human civilization, have invoked their consensus as an evidentiary ground for political action. These same scientists have also used their claim of a consensus to shame opposing points of view (climate change skeptics) as coming from “climate change deniers.”

Consensus, or “general acceptance” as it is sometimes cast in legal discussions, is rarely more than nose counting. At best, consensus is a proxy for data quality and inferential validity. At worst, consensus is a manifestation of group think and herd mentality. Debates about climate change, as well as most scientific issues, would progress more dependably if there were more data, and less harrumphing about consensus.

Olah’s Nobel Speech

One Nobel laureate, Professor George Olah, explicitly rejected the kumbaya view of science and its misplaced emphasis on consensus and collaboration. In accepting his Nobel Prize in Chemistry, Olah emphasized the value of adversarial challenges in refining and establishing scientific discovery:

“Intensive, critical studies of a controversial topic always help to eliminate the possibility of any errors. One of my favorite quotation is that by George von Bekessy (Nobel Prize in Medicine, 1961).

‘[One] way of dealing with errors is to have friends who are willing to spend the time necessary to carry out a critical examination of the experimental design beforehand and the results after the experiments have been completed. An even better way is to have an enemy. An enemy is willing to devote a vast amount of time and brain power to ferreting out errors both large and small, and this without any compensation. The trouble is that really capable enemies are scarce; most of them are only ordinary. Another trouble with enemies is that they sometimes develop into friends and lose a good deal of their zeal. It was in this way the writer lost his three best enemies. Everyone, not just scientists, needs a few good enemies!’”

George A. Olah, “My Search for Carbocations and Their Role in Chemistry,” Nobel Lecture (Dec. 8, 1994), quoting George von Békésy, Experiments in Hearing 8 (N.Y. 1960); see also McMillan v. Togus Reg’l Office, Dep’t of Veterans Affairs, 294 F. Supp. 2d 305, 317 (E.D.N.Y. 2003) (“As in political controversy, ‘science is, above all, an adversary process.’”) (internal citation omitted).

Carl Sagan expressed similar views about the importance of skepticism in science :

“At the heart of science is an essential balance between two seemingly contradictory attitudes — an openness to new ideas, no matter how bizarre or counterintuitive they may be, and the most ruthless skeptical scrutiny of all ideas, old and new. This is how deep truths are winnowed from deep nonsense.”

Carl Sagan, The Demon-Haunted World: Science as a Candle in the Dark (1995); See also Cary Coglianese, “The Limits of Consensus,” 41 Environment 28 (April 1999).

Michael Crichton, no fan of Sagan, agreed at least on the principle:

“I want to . . . talk about this notion of consensus, and the rise of what has been called consensus science. I regard consensus science as an extremely pernicious development that ought to be stopped cold in its tracks. Historically, the claim of consensus has been the first refuge of scoundrels; it is a way to avoid debate by claiming that the matter is already settled. Whenever you hear the consensus of scientists agrees on something or other, reach for your wallet, because you’re being had.

Let’s be clear: the work of science has nothing whatever to do with consensus. Consensus is the business of politics. Science, on the contrary, requires only one investigator who happens to be right, which means that he or she has results that are verifiable by reference to the real world. In science consensus is irrelevant. What is relevant is [sic] reproducible results. The greatest scientists in history are great precisely because they broke with the consensus.  There is no such thing as consensus science. If it’s consensus, it isn’t science. If it’s science, it isn’t consensus. Period.34

Michael Crichton, “Lecture at California Institute of Technology: Aliens Cause Global Warming” (Jan. 17, 2003) (describing many examples of how “consensus” science historically has frustrated scientific progress).

Crystalline Silica, Carcinogenesis, and Faux Consensus

Clearly, there are times when consensus in science works against knowledge and data-driven inferences. Consider the saga of crystalline silica and lung cancer. Suggestions that silica causes lung cancer date back to the 1930s, but the suggestions were dispelled by data. The available data were evaluated by the likes of Wilhelm Heuper[1], Cuyler Hammond[2] (Selikoff’s go-to-epidemiologist), Gerrit Schepers[3], and Hans Weill[4]. Even etiologic fabulists, such as Kaye Kilburn, disclaimed any connection between silica or silicosis and lung cancer[5]. As recently as 1988, august international committees, writing for the National Institute of Occupational Safety and Health, acknowledged the evidentiary insufficiency of any claim that silica caused lung cancer[6].

IARC (1987)

So what happened to the “consensus”? A group of activist scientists, who disagreed with the consensus, sought to establish their own, new consensus. Working through the International Agency for Research on Cancer (IARC), these scientists were able to inject themselves into the IARC working group process, and gradually raise the IARC ranking of crystalline silica. In 1987, the advocate scientists were able to move the IARC to adopt a “limited evidence” classification for crystalline silica.

The term “limited evidence” is defined incoherently by the IARC as evidence that provides for a “credible” causal explanation, even though chance, bias, and confounding have not been adequately excluded. Despite the incoherent definition that giveth and taketh away, the 1987 IARC reclassification[7] into Group 2A had regulatory consequences that saw silica classified as a “regulatory carcinogen,” or a substance that was “reasonably anticipated to be a carcinogen.”

The advocates’ prophecy was self-fulfilling. In 1996, another working group of the IARC met in Lyon, France, to deliberate on the classification of crystalline silica. The 1996 working group agreed, by a close vote, to reclassify crystalline silica as a “known human carcinogen,” or a Group 1 carcinogen. The decision was accepted and reported officially in volume 68 of the IARC monographs, in 1997.

According to participants, the debate was intense and the vote close. Here is the description from one of the combatants;

“When the IARC Working Group met in Lyon in October 1996 to assess the carcinogenicity of crystalline silica, a seemingly interminable debate ensued, only curtailed by a reminder from the Secretariat that the IARC was concerned with the identification of carcinogenic hazards and not the evaluation of risks. The important distinction between the potential to cause disease in certain circumstances, and in what circumstances, is not always appreciated.

*   *   *   *   *

Even so, the debate in Lyon continued for some time, finally ending in a narrow vote, reflecting the majority view of the experts present at that particular time.”

See Corbett McDonald, “Silica and Lung Cancer: Hazard or Risk,” 44 Ann. Occup. Hyg. 1, 1 (2000); see also Corbett McDonald & Nicola Cherry, “Crystalline Silica and Lung Cancer: The Problem of Conflicting Evidence,” 8 Indoor Built Env’t 8 (1999).

Although the IARC reclassification hardly put the silica lung cancer debate to rest, it did push the regulatory agencies to walk in lockstep with the IARC and declare crystalline silica to be a “known human carcinogen.” More important, it gave regulators and scientists an excuse to avoid the hard business of evaluating complicated data, and of thinking for themselves.

Post IARC

From a sociology of science perspective, the aftermath of the 1997 IARC monograph is a fascinating natural experiment to view the creation of a sudden, thinly supported, new orthodoxy. To be sure, there were scientists who looked carefully at the IARC’s stated bases and found them inadequate, inconsistent, and incoherent[8]. One well-regarded pulmonary text in particular gives the IARC and regulatory agencies little deference:

“Silica-induced lung cancer

A series of studies suggesting that there might be a link between silica inhalation and lung cancer was reviewed by the International Agency for Research on Cancer in 1987, leading to the conclusion that the evidence for carcinogenicity of crystalline silica in experimental animals was sufficient, while in humans it was limited.112 Subsequent epidemiological publications were reviewed in 1996, when it was concluded that the epidemiological evidence linking exposure to silica to the risk of lung cancer had become somewhat stronger.113 but that in the absence of lung fibrosis remained scanty.113 The pathological evidence in humans is also weak in that premalignant changes around silicotic nodules are seldom evident.114 Nevertheless, on this rather insubstantial evidence, lung cancer in the presence of silicosis (but not coal or mixed-dust pneumoconiosis) has been accepted as a pre­scribed industrial disease in the UK since 1992.115 Some subsequent studies have provided support for this decision.116 In contrast to the sparse data on classic silicosis, the evidence linking carcinoma of the lung to the rare diffuse pattern of fibrosis attributed to silica and mixed dusts is much stronger and appears incontrovertible.33,92”

Bryan Corrin[9] & Andrew Nicholson, Pathology of the Lungs (3d ed. 2011).

=======================================================

Cognitive biases cause some people to see a glass half full, while others see it half empty. Add a “scientific consensus” to the mix, and many people will see a glass filled 5% as 95% full.

Consider a paper by Centers for Disease Control and NIOSH authors on silica exposure and morality from various diseases. Geoffrey M. Calvert, Faye L. Rice, James M. Boiano, J. W. Sheehy, and Wayne T. Sanderson, “Occupational silica exposure and risk of various diseases: an analysis using death certificates from 27 states of the United States,” 60 Occup. Envt’l Med. 122 (2003). The paper was nominated for the Charles Shepard Award for Best Scientific Publication by a CDC employee, and was published in the British Medical Journal’s publication on occupational medicine. The study analyzed death certificate data from the U.S. National Occupational Mortality Surveillance (NOMS) system, which is based upon the collaboration of NIOSH, the National Center for Health Statistics, the National Cancer Institute, and some state health departments. Id. at 122.

From about 4.8 million death certificates included in their analysis, the authors found a statistically decreased mortality odds ratio (MOR) for lung cancer among those who had silicosis (MOR = 0.70, 95% C.I., 0.55 to 0.89). Of course, with silicosis on the death certificates along with lung cancer, the investigators could be reasonably certain about silica exposure. Given the group-think in occupational medicine about silica and lung cancer, the authors struggled to explain away their finding:

“Although many studies observed that silicotics have an increased risk for lung cancer, a few studies, including ours, found evidence suggesting the lack of such an association. Although this lack of consistency across studies may be related to differences in study design, it suggests that silicosis is not necessary for an increased risk of lung cancer among silica exposed workers.”

Well this statement is at best disingenuous. The authors did not merely find a lack of an association; they found a statistically significance inverse or “negative” association between silicosis and lung cancer. So it is not the case that silicosis is not necessary for an increased risk; silicosis is antithetical to an increased risk.

Looking at only death certificate information, without any data on known or suspected confounders (“diet, hobbies, tobacco use, alcohol use, or medication,” id. at 126, or comorbid diseases or pulmonary impairment, or other occupational or environmental exposures), the authors inferred low, medium, high, and “super high” silica exposure from job categories. Comparing the ever-exposed categories with low exposure yielded absolutely no association between exposure and lung cancer, and subgroup analyses (without any correction for multiple comparisons) found little association, although two subgroups were nominally statistically significantly increased, and one was nominally statistically significantly decreased, at very small deviations from expected:

Lung Cancer Mortality Odds Ratios (p-value for trend < 0.001)

ever vs. low/no exposure:                0.99 (0.98 to 1.00)

medium vs. low/no exposure:         0.88 (0.87 to 0.90)

high vs. low/no exposure:                 1.13 (1.11 to 1.15)

super high vs. low/no exposure:      1.13 (1.06 to 1.21)

Id. at Table 4, and 124.

On this weak evidentiary display, the authors declare that their “study corroborates the association between crystalline silica exposure and silicosis, lung cancer.” Id. at 123. In their conclusions, they elaborate:

“Our findings support an association between high level crystalline silica exposure and lung cancer. The statistically significant MORs for high and super high exposures compared with low/no exposure (MORs = 1.13) are consistent with the relative risk of 1.3 reported in a meta-analysis of 16 cohort and case-control studies of lung cancer in crystalline silica exposed workers without silicosis”

Id. at 126. Actually not; Calvert’s reported MORs exclude an OR of 1.3.

The Calvert study thus is a stunning example of authors, prominent in the field of public health, looking at largely exculpatory data and declaring that they have confirmed an important finding of silica carcinogenesis. And to think that United States taxpayers paid for this paper, and that the authors almost received an honorific award for this thing!

[1] Wilhelm Hueper, “Environmental Lung Cancer,” 20 Industrial Medicine & Surgery 49, 55-56 (1951) (“However, the great majority of investigators have come to the conclusion that there does not exist any causal relation between silicosis and pulmonary or laryngeal malignancy”).

[2] Cuyler Hammond & W. Machle, “Environmental and Occupational Factors in the Development of Lung Cancer,” Ch. 3, pp. 41, 50, in E. Mayer & H. Maier, Pulmonary Carcinoma: Pathogenesis, Diagnosis, and Treatment (N.Y. 1956) (“Studies by Vorwald (41) and others agree in the conclusion that pneumoconiosis in general, and silicosis in particular, do not involve any predisposition of lung cancer.”).

[3] Gerrit Schepers, “Occupational Chest Diseases,” Chap. 33, in A. Fleming, et al., eds., Modern Occupational Medicine at 455 (Philadelphia 2d ed. 1960) (“Lung cancer, of course, occurs in silicotics and is on the increase. Thus far, however, statistical studies have failed to reveal a relatively enhanced incidence of pulmonary neoplasia in silicotic subjects.”).

[4] Ziskind, Jones, and Weill, “State of the Art: Silicosis” 113 Am. Rev. Respir. Dis. 643, 653 (1976) (“There is no indication that silicosis is associated with increased risk for the development of cancer of the respiratory or other systems.”); Weill, Jones, and Parkes, “Silicosis and Related Diseases, Chap. 12, in Occupational Lung Disorders (3d ed. 1994) (“It may be reasonably concluded that the evidence to date that occupational exposure to silica results in excess lung cancer risk is not yet persuasive.”).

[5] Kaye Kilburn, Ruth Lilis, Edwin Holstein, “Silicosis,” in Maxcy-Rosenau, Public Health and Preventive Medicine, 11th ed., at 606 (N.Y. 1980) (“Lung cancer is apparently not a complication of silicosis”).

[6] NIOSH Silicosis and Silicate Disease Committee, “Diseases Associated With Exposure to Silica and Nonfibrous Silicate Minerals,” 112 Arch. Path. & Lab. Med. 673, 711b, ¶ 2 (1988) (“The epidemiological evidence at present is insufficient to permit conclusions regarding the role of silica in the pathogenesis of bronchogenic carcinoma.”)

[7] 42 IARC Monographs on the Evaluation of the Carcinogenic Risk of Chemicals to Humans at 22, 111, § 4.4 (1987).

[8] See, e.g., Patrick A. Hessel, John F. Gamble, J. Bernard L. Gee, Graham Gibbs, Francis H. Y. Green, W. Keith C. Morgan, and Brooke T. Mossman, “Silica, Silicosis, and Lung Cancer: A Response to A Recent Working Group,” 42 J. Occup. Envt’l Med. 704, 718 (2000) (“The data demonstrate a lack of association between lung cancer and exposure to crystalline silica in human studies. Furthermore, silica is not directly genotoxic and has been to be a pulmonary carcinogen in only one animal species, the rat, which seems to be an inappropriate carcinogenesis in humans.”)

[9] Professor of Thoracic Pathology, National Heart and Lung Institute, Imperial College School of Medicine; Honorary Consultant Pathologist, Brompton Hospital, London, UK.

Posted in Causation, Cognitive Biases, Scientific Evidence | Comments Off on Science as Adversarial Process versus Group Think

Newer Entries »