Two weeks ago, I attended a (very good) workshop on experimental political science. The keynote, given by Neil Malhotra, was titled
Are We At Peak Experiment? Confronting Publication Bias and Research Transparency in Political Science Experiments
The talk adressed publication bias, the need for replication studies and similar issues - but he avoided one critical issue: Fraud.
Economists - the old school, neoclassical type - talk of moral hazard problems, and assume that people are tempted to cheat, especially when stakes are high and the probability of being caught is low. Recommendations to minimize these problems typically include transparency, checks and punishment. But when transparency is necessarily low due to information asymmetries, and when checks and punishment are costly, there will be some cheating.
This reasoning is completely standard when is comes to, for example, tax evasion, the take-up of welfare benefits or corruption. But it also applies to economists themselves - and in this case, to experimental political scientists LaCour and Green, authors of the now retracted study
LaCour, Michael J. and Donald P. Green. 2014. When contact changes minds: An experiment on transmission of support for gay equality. Science 346(6215): 1366-1369.
Research fraud in connection with experiments has already been exposed in psychology several times: Diederik Stapel and Marc Hauser. (Have there been any major scandals in experimental economics yet?)
So, what's next? I see three scenarios:
Worst case scenario:
- More fraud is discovered.
- Credibility of experimental research falls.
- We were indeed at peak experiment.
More likely scenario:
- The discovery of some fraud cases has a disciplining effect.
- Replications are encouraged and rewarded much more than today.
- Perhaps it is finally time for Journal of Robustness Analysis and Replication Studies?
What one could hope for:
- A separation of experiment design and experiment implementation.
- Less focus on "statistical significance"
- Some acknowledgement of the upside of doing research with data that anyone can easily download and verify
- Less whining about neoclassical economics being unable to predict human behavior