Explanation of Payouts

Each round has both surveys and markets on the same base claims. Markets pay more, later, while surveys pay less, sooner. Markets pay out proportionally to winning shares. Surveys pay fixed amounts to top ranks in each batch.  (They are also different forecasting methods, and we will be comparing strengths and weaknesses.)

Markets pay based on actual outcome once known, and so are worth twice as much in total. However, we expect only 100 to be tested by direct replication, and will not know which until DARPA reveals them, probably mid-2020.  

Surveys pay each round, using a peer prediction mechanism with bias correction to rank each participant for each of their forecasts. These peer prediction scores are computed based on 

  1. an inferred distribution of the true outcome using all contributed predictions 
  2. a bias estimation and correction procedure to correct the scoring step.


With high probability, the top rankings will track ground truth. But we won’t know until the contest ends. Therefore these are worth less. 

Details in the rules, and summarized below.

Round 0:
  • Round 0 has only 12 claims.
  • Starting on August 12, the Round 0 Survey has a $960 prize payout as: 6x$80, 6x$40, and 12x$20, to the top 24 forecasters. To participate in the Survey, forecasters must be registered by 10:00 pm UTC on Sunday, August 11. Payouts are expected at the end of September.
  • Starting on August 19, The Round 0 Market will pay on average about $360 per resolving claim ($4,320 total), paid proportionally by winning shares in the Round, when all answers are known (July  2020). All 12 of these claims are expected to resolve. If you have 5% of the winning shares, in any combination of claims, you earn $216.
Rounds 1 – 10:
  • Starting September 2019, RM will  predict replication for 3,000 claims in 4-week  rounds of about 300 claims.  In each round there will be a week of surveys followed by two weeks of markets.
  • Surveys are batches of 10 claims,  all from the same journal where possible. Participants must finish a survey before doing another. Participants may complete as many surveys as they want. Each batch will pay $160 to the Top 4 participants as: $80, $40, $20, $20 for a total of $4,800 in survey prizes per round. Surveys pay out at the end of each round (after markets close).
  • Markets on all 300 claims open shortly after the surveys close. Trading is open for two weeks, then the market is frozen until the final result is known. Each round only about 10  markets will resolve, and we will not know which until DARPA reveals that to us (possibly not until mid-2020).
  • The total prize amount for a Round is the number of resolving claims on that Round * $900: markets pay $900 per on resolution, that total will be divided proportionally among all winning shares. The number of shares, and therefore the associated payouts may vary by claim.
  • At least $12K in prize money has been reserved for adjustments or new incentives to be announced after January.

Want to receive updates about Replication Markets? Share your contact information below.

© 2019 All rights Reserved

We’re sorry to see you go! Please visit our social media sites.

This site uses cookies to provide you with a better browsing experience.

Visit our Privacy Policy for more information.