Will COVID-19 studies replicate?

Replication Markets recently estimated replication probabilities for 100 COVID-19 studies from the social and behavioral sciences. We don’t have all the results yet, but let’s see what our forecasters thought. (Note: We just launched another COVID-19 project, we hope you’ll sign-up to participate or learn more).

Reviewing our Previous COVID-19 Forecasts

Big picture
  • ~75% of studies (everything to the right of the blue line) are estimated to have a better than 50% chance of replicating successfully.
  • In prior research only ~50% of studies replicate successfully, so it’s interesting to note that our forecasters are more optimistic about the studies in this sample.
  • Comparing Surveys and Markets Participants in Replication Markets express their beliefs about the probability of successful replication two ways: with prediction surveys and markets. We’ll compare the forecasts we get from each of these methods
  • The market estimates are slightly lower (by 1.9%)
  • The market estimates are more varied (ratio of variances = 1.5)
  • Fig 1: Note that each point corresponds to the estimated replication probability for a different study.
    Rplot01

    Do surveys and markets agree about individual studies?

    • The plot below indicates a fair amount of agreement on replication probabilities between the surveys and the market.
    • The points above the line are studies for which the survey was more optimistic than the market.
    • We hope combining surveys and markets can improve the accuracy of our forecasts

    Are these forecasts accurate?

    We evaluate our replication forecasts with replication results. As of today we have only 8 replication results. Of the 100 we forecast, we expect 20-50 to eventually receive a replication efforts (as part of our research program, DARPA SCORE).

    • All 8 studies replicated successfully, a positive surprise!
    • We assigned > 0.6 probability of replication success to each successful replication.
    • Despite being 8 for 8, we are certain our forecasts are imperfect. The 8 studies replicated look “more obvious” than most. We will learn more about where we were right and wrong over time as more replication results arrive.
    market_vs_survey_forecasts_scatterplot

    If you find this interesting, please sign-up to participate or learn more.

    Note: Cross-posted on RM’s SCORE project blog [LINK]

    Mike Bishop

    There is NO COST to participate in Replication Markets.

    This research is supported (in part) by the Fetzer Franklin Fund of the John E. Fetzer Memorial Trust. It uses a platform developed for DARPA SCORE, and some staff are supported by SCORE while working on this. We are grateful for their support.

    We’re sorry to see you go! Please visit our social media sites.

    This site uses cookies to provide you with a better browsing experience.

    Visit our Privacy Policy for more information.