Washington, D.C. 20460
Subject: SAB Review of EPA’s Retrospective Cost Study of the Costs of EPA Regulations: An Interim Report of Five Case Studies (March 2012) Dear Acting Administrator Perciasepe:
EPA’s Science Advisory Board (SAB) was asked to review and comment on the Retrospective Cost Study of the Costs of EPA Regulations: An Interim Report of Five Case Studies (March 2012, RCS). The RCS is composed of five case studies developed by EPA’s National Center for Environmental Economics (NCEE) to investigate how well EPA has predicted the costs of regulatory compliance. The RCS sought to compare cost estimates ex post (after the regulation) to EPA’s ex ante (before the regulation) predictions. EPA asked the SAB a series of questions on the case study approach applied in the RCS and specifically on appropriate methodologies to be applied in these and future case studies.
The SAB applauds EPA’s efforts to evaluate the ex post costs of regulations and to examine systematic differences between the ex post and ex ante estimates. Ensuring that costs of regulations are accurately predicted and assessed is critical for designing cost-effective regulations. The RCS provided some much-needed attention to the topic of cost of regulatory compliance.
The SAB finds that the case studies in the RCS are useful in showing that there are often significant divergences in assumptions underlying the ex ante analysis and ex post outcomes regarding methods of compliance, timing of compliance, and impact of compliance on production and market parameters. A comprehensive assessment of the ex post costs of regulations is, however, limited by available data. In addition, there is significant heterogeneity across regulations in terms of the pollution medium, industry structure and size, timelines, variety of compliance methods and rates of technological change. EPA’s case studies generally showed more evidence of overestimation of ex ante costs than underestimation although the results cannot be generalized given the small sample size of cases considered and paucity of data. Based on these case studies, the SAB agrees with the Agency’s conclusion that it is not possible to draw meaningful conclusions regarding general tendencies to under- or over-estimate costs in ex ante evaluations.
The SAB would like to see EPA broaden its goal beyond the narrow question of whether ex ante costs are higher or lower than ex post costs. Rather than simply focusing on the question of whether EPA generally overestimates or underestimates costs, the SAB recommends a focus on the drivers of the differences in ex post and ex ante costs so that insights can be gained that would improve ex-ante cost estimation in the future. To do this, a more systematic approach needs to be applied across case studies. This approach should be based on an overarching conceptual framework that provides a foundation for the RCS and explains the various reasons that could lead to differences between ex post and ex ante costs. This framework can be used to develop hypotheses whose validity can be examined in specific cases. For example, one hypothesis could be that EPA regulations induce innovation in industry and drive down costs of compliance over time. Case studies can examine the extent to which this hypothesis is supported for various regulations. EPA can then use this insight to change their assumptions about technological change in ex-ante cost of compliance estimation.
More case studies are needed. Reliance on a small number of case studies is not adequate to support the task. EPA’s sample size will need to extend far beyond the 5 case studies completed and 5 case studies planned. These case studies should be selected randomly rather than be ones where differences are expected to be large or more data is available in order to avoid selection bias and to allow for more generalizable conclusions of the causes of differences between ex post and ex ante costs. The SAB recommends that EPA do more, but shorter qualitative analyses of regulation, focusing more on qualitative factors with the goal of presenting well-formed and documented ‘stories’ that identify the large drivers of differences between ex ante and ex post results.
More data are needed. The Agency does not typically collect information on costs of compliance ex post. As a result, EPA was limited to publicly available data and collaboration with industry experts. If the Agency is to draw meaningful lessons for future ex ante cost estimates as well as for the direction and extent to which ex post cost estimates diverge from ex ante estimates, it must go beyond the existing data sources and pursue a primary data collection approach using surveys, focus groups and broad based expert opinions as well as consult with the outside research community. The SAB recommends that EPA consider ways of building in a routine ex post data collection effort either within the Agency or in collaboration with other agencies and strongly urges the Agency to invest the resources needed to do a proper evaluation. This data would be useful to improve ex ante cost analysis and increase the cost-effectiveness of future regulations with potential savings in the costs of compliance.
The enclosed SAB report offers specific suggestions for how EPA’s methodology might be improved. We appreciate the opportunity to provide advice on this important subject and look forward to the Agency’s response.