Somehow, before the Supreme Court breathed life into Federal Rule of Evidence 702, parties sometimes found a way to challenge dubious scientific evidence in court. One good example is the challenge to the United States Environmental Protection Agency’s risk assessment of passive smoking, also known as environmental tobacco smoke (ETS). In 1992, the Environmental Protection Agency (EPA) published a risk assessment of lung cancer (and other) risks from ETS. See Respiratory Health Effects of Passive Smoking: Lung Cancer and Other Disorders EPA/600/6-90/006F (1992). The agency concluded that ETS causes about 3,000 lung cancer deaths each year among non-smoking adults in the United States. See also EPA “Fact Sheet: Respiratory Health Effects of Passive Smoking,” Office of Research & Development; EPA Document Number 43-F-93-003 (Jan. 1993).
Various plaintiffs, including tobacco companies, challenged the EPA’s conclusions as agency action that violated administrative and statutory procedures. The plaintiffs further claimed that the EPA had manufactured its methods to achieve the result it desired in advance of the analyses. In other words, plaintiffs asserted that the EPA’s issuance of the ETS report violated the Administrative Procedures Act’ procedural requirements, as well as the requirements of the specific enabling legislation, the Radon Gas and Indoor Air Quality Research Act, Pub.L. No. 99–499, 100 Stat. 1758–60 (1986) (codified at 42 U.S.C. § 7401 note (1994)). A federal district court agreed with the methodological challenges to the EPA’s report, but the Court of Appeals reversed on grounds that the agency’s report was not reviewable agency action. Flue-Cured Tobacco Cooperative Stabilization Corp. v. EPA, 4 F. Supp. 2d 435 (M.D.N.C. 1998), rev’d on other grounds, 313 F.3d 852, 862 (4th Cir. 2002) (Widener, J.) (holding that the issuance of the report was not “final agency action”). The district court’s assessment of the validity issues were not addressed by the appellate court.
Notwithstanding the district court’s findings, the EPA continues to claim that it had reached valid scientific conclusions using a “scientific approach”:
“EPA reached its conclusions concerning the potential for ETS to act as a human carcinogen based on an analysis of all of the available data, including more than 30 epidemiologic (human) studies looking specifically at passive smoking as well as information on active or direct smoking. In addition, EPA considered animal data, biological measurements of human uptake of tobacco smoke components and other available data. The conclusions were based on what is commonly known as the total weight-of-evidence” rather than on any one study or type of study.
The finding that ETS should be classified as a Group A carcinogen is based on the conclusive evidence of the dose-related lung carcinogenicity of mainstream smoke in active smokers and the similarities of mainstream and sidestream smoke given off by the burning end of the cigarette. The finding is bolstered by the statistically significant exposure-related increase in lung cancer in nonsmoking spouses of smokers which is found in an analysis of more than 30 epidemiology studies that examined the association between secondhand smoke and lung cancer.”
EPA “Fact Sheet: Respiratory Health Effects of Passive Smoking,” Office of Research and Development; EPA Document Number 43-F-93-003, January 1993 (emphasis added).
A prominent feature of the EPA’s analysis was a meta-analysis of epidemiologic studies of ETS and lung cancer. Interestingly, the tobacco industry plaintiffs did not appear to challenge the legitimacy of the basic meta-analytic enterprise, which was still controversial at the time. See, e.g., “Samuel Shapiro, Meta-analysis/Smeta-analysis,” 140 Am. J. Epidem. 771 (1994); Alvan Feinstein, “Meta-Analysis: Statistical Alchemy for the 21st Century,” 48 J. Clin. Epidem. 71 (1995). Their challenge went straight to the validity of the EPA’s meta-analysis, and a documented post hoc change in the agency’s statistical plan for analyzing the meta-analysis results. Only a few years earlier, the defense in polychlorobiphenyl (PCB) litigation broadly challenged a plaintiffs’ expert witness’s use of meta-analysis of observational epidemiologic studies, only to have the Third Circuit reject the challenge and to direct the district court to review the validity of the meta-analysis as conducted by the witness. In re Paoli RR Yard PCB Litig., 706 F. Supp. 358, 373 (E.D. Pa. 1988), rev’d, 916 F.2d 829, 856-57 (3d Cir. 1990), cert. denied, 499 U.S. 961 (1991); see also Hines v. Consol. Rail Corp., 926 F.2d 262, 273 (3d Cir. 1991).
The EPA report was not the first attempt to use meta-analysis for the epidemiology of ETS and lung cancer. In 1986, the National Academy of Sciences reported a meta-analysis on the subject. See National Research Council, National Academy of Sciences, Environmental tobacco smoke: measuring exposures and assessing health effects (Wash. DC 1986). This earlier meta-analysis was also controversial. Indeed, some of the early concerns over the use of meta-analysis for observational epidemiologic studies arose in the context of studies of ETS. See, e.g., Joseph L. Fleiss & Alan J. Gross, “Meta-Analysis in Epidemiology, with Special Reference to Studies of the Association between Exposure to Environmental Tobacco Smoke and Lung Cancer: A Critique,” 44 J. Clin. Epidem. 127 (1991) (criticizing the National Research Council 1986 meta-analysis of ETS and lung cancer studies as unwarranted based upon the low quality of the studies included). These concerns were heightened by politicized use of meta-analyses in regulatory agencies to overclaim scientific conclusions from weak, inconclusive data.
In the EPA’s meta-analysis, statistical significance was achieved only by changing the criterion of significance, post hoc, from a two-tailed to a one-tailed 5% test. Perhaps more disturbing was the scientific gerrymandering that took place as to which studies to include and exclude from the meta-analysis.
In its first review of the EPA’s draft report, a committee of the agency’s Scientific Advisory Board, the IAQC [the Indoor Air Quality/Total Human Exposure Committee] found that the EPA’s ETS risk assessment violated one of the necessary criteria for a valid meta-analysis – a “precise definition of criteria used to include (or exclude) studies.” 4 F. Supp. 2d at 459 (citing EPA, An SAB Report: Review of Draft Environmental Tobacco Smoke Health Effects Document, EPA/SAB/IAQC/91/007 at 32–33 (1991) (SAB 1991 Review) (JA 9,497–98)). The agency had not provided specific criteria for including studies. The IAQC also noted that it was important to evaluate the consequences of having excluded studies in the form of sensitivity studies. In a later review, in 1992, both the EPA and the IAQC dropped this critique of the agency’s meta-analysis, without explanation. Id. at 459.
By the time the EPA released its ETS report in 1993, there were about 58 published epidemiologic studies available for inclusion in any meta-analysis. The EPA included only 31. The agency limited its analysis to nonsmoking women married to smoking spouses. There were 33 studies of this exposed group; the EPA included 31 of the 33. There were also available 12 studies of women exposed to ETS in their workplace, and 13 studies of women who were exposed to ETS as children. Id. at 458. There were three late-breaking studies of women with spousal exposures, but the EPA excluded two, without explanation. Id. at 459.
In reviewing the plaintiffs’ challenge, the district court noted that the EPA had given a bare, unconvincing explanation for excluding the childhood and workplace studies. Id. The EPA argued that there was less data in the childhood and workplace studies, but this assertion struck the court as an evasive rationale when one of the purposes of conducting a meta-analysis was to incorporate the data from smaller, less powerful studies. Id. 458-59. The primary author of the disputed chapter of the EPA report, Kenneth Brown, called the disputed studies “inadequate,” without providing a rational basis or explanation. The IAQC, in its earlier review of a 1991 draft report, recognized that the excluded studies provided less information, but concluded that the agency’s “the report should review and comment on the data that do exist… .” Id. at 459.
The court found the EPA’s selection of studies for inclusion in a meta-analysis to be “disturbing”:
“First, there is evidence in the record supporting the accusation that EPA ‘cherry picked’ its data. Without criteria for pooling studies into a meta-analysis, the court cannot determine whether the exclusion of studies likely to disprove EPA’s a priori hypothesis was coincidence or intentional. Second, EPA’s excluding nearly half of the available studies directly conflicts with EPA’s purported purpose for analyzing the epidemiological studies and conflicts with EPA’s Risk Assessment Guidelines. See ETS Risk Assessment at 4–29 (“These data should also be examined in the interest of weighing all the available evidence, as recommended by EPA’s carcinogen risk assessment guidelines (U.S.EPA, 1986a) ….” (emphasis added)). Third, EPA’s selective use of data conflicts with the Radon Research Act. The Act states EPA’s program shall ‘‘gather data and information on all aspects of indoor air quality….’’ Radon Research Act § 403(a)(1) (emphasis added). In conducting a risk assessment under the Act, EPA deliberately refused to assess information on all aspects of indoor air quality.”
4 F. Supp. 2d at 460.
The court was no doubt impressed by the duplicity of the agency’s claim to have used a “total weight of the evidence” approach to the question of causality, and its censoring of the analysis in a way that appeared to game the result. Id. at 454 The EPA’s guidelines called for basing conclusions on all available evidence. EPA’s Guidelines for Carcinogen Risk Assessment, 51 Fed. Reg. 33,996, 33,999-34,000 (1986).
Using evidence selectively, with a post hoc adoption of a one-tailed test of statistical significance, the EPA reported a summary estimate of risk of 1.19, and categorized ETS as a “Group A” carcinogens. In most of its previous Group A classifications, the agency had based its decisions upon much higher relative risks. Indeed, the agency had rejected Group A classifications when relative risks were found to be less than three. 4 F. Supp. 2d at 461. The sum total of the agency’s methodological laxity was too much for the district court, which struck the chapters of the EPA report. Four years later, the Fourth Circuit of the U.S. Court of Appeals reversed, on grounds that the EPA report was not reviewable agency action.
The EPA report became a lightning rod for methodological criticism of meta-analysis for observational studies, and the EPA’s use of meta-analysis. Critics argued that the EPA had succumbed to political pressure from the anti-tobacco lobby. See, e.g., Gio B. Gori & John C. Luik, Passive Smoke: The EPA’s Betrayal of Science and Policy (Vancouver, BC: The Fraser Institute 1999); John C. Luik, “Pandora’s Box: The Dangers of Politically Corrupted Science for Democratic Public Policy,” Bostonia 54 (Winter 1999-94). See also Elizabeth Fisher, “Case law analysis. Passive smoking and active courts: the nature and role of risk regulators in the US and UK. Flue-cured Tobacco Co-op v US Environmental Protection Agency,” 12 J. Envt’l Law 79 (2000).
The federal government has been trying to defend the EPA’s 1992 report, ever since. In 1998, upon listing ETS as a known carcinogen, the Department of Health and Human Services noted that “[t]he individual studies were carefully summarized and evaluated” in the 1992 EPA report. U.S. Dep’t of Health & Human Services, National Toxicology Program, Final Report on Carcinogens – Background Document for Environmental Tobacco Smoke: Meeting of the NTP Board of Scientific Counselors – Report on Carcinogens Subcommittee at 24 (Research Triangle Park, NC 1998). Anti-tobacco scientists, including scientists involved in the EPA report, have attacked the motives of the industry, and of the scientists who have challenged the report. See, e.g., Jonathan M. Samet & Thomas A. Burke, “Turning Science Into Junk: The Tobacco Industry and Passive Smoking,” 91 Am. J. Pub. Health 1742 (2001); Monique E. Muggl, Richard D. Hurt, and James Repace, “The Tobacco Industry’s Political Efforts to Derail the EPA Report on ETS,” 26 Am. J. Prev. Med. 167 (2004); Deborah E. Barnes & Lisa A. Bero, “Why review articles on the health effects of passive smoking reach different conclusions,” 279 J. Am. Med. Ass’n 1566 (1998).
Of course, science did not remain status quo 1992. Later studies were published, and the controversy continued, such that the 1992 meta-analysis is now largely scientifically irrelevant. See James Enstrom & Geoffrey Kabat, “Environmental tobacco smoke and tobacco related mortality in a prospective study of Californians, 1960-98,” 326 Br. Med. J. 1057 (2003); G. Davey Smith, “Effect of passive smoking on health: More information is available, but the controversy still persists,” 326 Br. Med. J. 1048–9 (2003).
A troubling implication of those who attack the tobacco industry is that the industry was not allowed to raise methodological challenges to the EPA’s purported use of a scientific method. The EPA defenders rarely engage with the specifics of the methodological challenge or the district court’s review. Another implication is that the EPA’s meta-analysis remains a clear example of where a regulatory agency could have acted upon a precautionary principle, but chose to dress up its analysis as something it was not: a scientific conclusion of causality. Given that the agency was not even engaged in reviewable agency action, and that it had plenty of biological plausibility for a precautionary finding that ETS causes lung cancer, the agency could easily have avoided the vitriolic debate it engendered with its 1992 report.
