Replication Markets is a government funded research project to forecast the outcomes of thousands of potential social science replications to be done from mid-2019 to mid-2020. As part of the project, the Center for Open Science has selected 3,000 studies for us to forecast, from the last 10 years of published social science. They will select a representative sample (unknown to us) of 100+ studies for direct replication, and supervise those replications. The results will tell us if we have met or exceed the 70-80% accuracy of our previous replication markets, and inform Confidence Scores on the reliability of the claims themselves.
We are still awaiting final IRB approval and hope to begin forecasting in August. In the meantime, see the link at the bottom to follow the project.
Approach
Our approach will use both surveys and markets. Based on our previous work we expect the markets to have a small edge, but we are also working on smarter ways to aggregate the surveys.
Prediction markets and surveys are used to forecast the outcome of a wide variety of topics from elections to geopolitics to delivery schedules. A recent paper from the Good Judgment Team provides the best head-to-head comparison yet, and shows that clever averaging outperforms markets, which outperform simple averages. We will be testing the following improvements over our previous work:
- Better aggregation methods for the surveys.
- Starting the markets with informed prices, not just 50%.
- Allowing bots to trade in the market, with some rules of engagement.
- Allowing connections between claims (starting later in the year)
- Because only about 5% of the claims will resolve, we are using Decision Market scoring rules to preserve incentives. (See “Decision Markets with Good Incentives”.)
Once running, we expect to work in monthly batches of about 300 claims. Specifics will be described in the Informed Consent statement and related materials.
Follow Us
Although final results will not be available until mid-2020, you may follow the project to receive updates and relevant posts.
Replication Markets will be launched in August 2019 pending IRB approval; you can sign up here to be notified.
