Welcome to Replication Markets!

Replication Markets is a government funded research project to forecast the outcomes of thousands of potential social science replications to be done from mid-2019 to mid-2020. As part of the project, the Center for Open Science has selected 3,000 studies for us to forecast, from the last 10 years of published social science.  They will select a representative sample (unknown to us) of 100+ studies for direct replication, and supervise those replications. The results will tell us if we have met or exceed the 70-80% accuracy of our previous replication markets, and inform Confidence Scores on the reliability of the claims themselves.

We are still awaiting final IRB approval and hope to begin forecasting in August. In the meantime, see the link at the bottom to follow the project.

 

Approach

Our approach will use both surveys and markets. Based on our previous work we expect the markets to have a small edge, but we are also working on smarter ways to aggregate the surveys.  

Both of our approaches use crowdsourcing – because the collective wisdom of an informed and diverse group is often more accurate at forecasting the outcome of events than that of one individual.

Prediction markets and surveys are used to forecast the outcome of a wide variety of topics from elections to geopolitics to delivery schedules. A recent paper from the Good Judgment Team provides the best head-to-head comparison yet, and shows that clever averaging outperforms markets, which outperform simple averages. We will be testing the following improvements over our previous work:

  • Better aggregation methods for the surveys.
  • Starting the markets with informed prices, not just 50%.  
  • Allowing bots to trade in the market, with some rules of engagement.
  • Allowing connections between claims (starting later in the year)
  • Because only about 5% of the claims will resolve, we are using Decision Market scoring rules to preserve incentives.  (See “Decision Markets with Good Incentives”.)

Once running, we expect to work in monthly batches of about 300 claims. Specifics will be described in the Informed Consent statement and related materials.

 Follow Us

Although final results will not be available until mid-2020, you may follow the project to receive updates and relevant posts.

Replication Markets will be launched in August 2019 pending IRB approval; you can sign up here to be notified.

Contribute to the discussion...

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Want to receive updates about Replication Markets? Share your contact information below.

We’re sorry to see you go! Please visit our social media sites.

This site uses cookies to provide you with a better browsing experience.

Visit our Privacy Policy for more information.