Of Forking Paths and Tied Hands: Selective Publication of Findings, and What Economists Should Do about It
Journal of Economic Perspectives
vol. 35,
no. 3, Summer 2021
(pp. 175-92)
(Complimentary)
Abstract
A key challenge for interpreting published empirical research is the fact that published findings might be selected by researchers or by journals. Selection might be based on criteria such as significance, consistency with theory, or the surprisingness of findings or their plausibility. Selection leads to biased estimates, reduced coverage of confidence intervals, and distorted posterior beliefs. I review methods for detecting and quantifying selection based on the distribution of p-values, systematic replication studies, and meta-studies. I then discuss the conflicting recommendations regarding selection resulting from alternative objectives, in particular, the validity of inference versus the relevance of findings for decision-makers. Based on this discussion, I consider various reform proposals, such as deemphasizing significance, pre-analysis plans, journals for null results and replication studies, and a functionally differentiated publication system. In conclusion, I argue that we need alternative foundations of statistics that go beyond the single-agent model of decision theory.Citation
Kasy, Maximilian. 2021. "Of Forking Paths and Tied Hands: Selective Publication of Findings, and What Economists Should Do about It." Journal of Economic Perspectives, 35 (3): 175-92. DOI: 10.1257/jep.35.3.175Additional Materials
JEL Classification
- A14 Sociology of Economics
- C80 Data Collection and Data Estimation Methodology; Computer Programs: General
- L82 Entertainment; Media
There are no comments for this article.
Login to Comment