Is proprietary data the biggest threat to economic research credibility?

Question

I spoke with Ted Miguel in today’s AEA research highlight about the reliability of economics research. He’s generally optimistic, but thinks proprietary data threatens credibility more than anything else. Do you agree? How do you think this should be addressed?

Here's a link to the interview: https://www.aeaweb.org/research/transparency-reproducibility-credibility-economics.

Comments

I wonder if two of the major ways in which proprietary data might cut off credibility - inability to reproduce using the original data, and opacity of analysis - are things that we don't tend to check all that often anyway (there's a recent Hamermesh paper with actual numbers). i.e. maybe it's not great but I wonder how much worse it really is than the status quo

Answer 1

This is a challenging problem.  The most natural solution might be requiring something like a GitHub style log or history of all of the specifications some has run.  Or to pre-register quasi experiments and specifications.  

The challenge is whther people can circumvent these steps by p hacking first and then pregistering after they know what works.  If cheating in sports and PED tells  us anything it's that cheaters can stay one step ahead of enforcement.   

However maybe these steps still would help guide unintentional cheaters who want to be honest but get sucked into looking for stars.

Answer 2

Some journals like the Econometrics Journal https://www.res.org.uk/journals/the-econometrics-journal/submissions/replication-policy.html discourage use of proprietary data. It should only be accepted if a convincing explanation is provided why such data needs to be used to answer the research question and if it is clearly documented how others can obtain the data. That should be a general standard also followed by the AEA, for example.