Payouts – Replication Markets

Explanation of Payouts

Each round has both surveys and markets on the same base claims. Markets pay more, later, while surveys pay less, sooner. Markets pay out proportionally to winning shares. Surveys pay fixed amounts to top ranks in each batch. (They are also different forecasting methods, and we will be comparing strengths and weaknesses.)

Markets pay based on actual outcome once known, and so are worth more. We will not know which claims resolve until DARPA reveals them, probably ~~mid-2020~~ by 30 November 2020.

Surveys pay each round, using a peer prediction mechanism with bias correction to rank each participant for each of their forecasts. These peer prediction scores are computed based on

an inferred distribution of the true outcome using all contributed predictions
a bias estimation and correction procedure to correct the scoring step.

With high probability, the top rankings will track ground truth. But we won’t know until the contest ends. Therefore these are worth less.

Details in the rules, and summarized below.

Round 0:

Round 0 has only 12 claims.
Starting on August 12, the Round 0 Survey has a $960 prize payout as: 6x$80, 6x$40, and 12x$20, to the top 24 forecasters. To participate in the Survey, forecasters must be registered by 10:00 pm UTC on Sunday, August 11. Payouts are expected at the end of September.
Starting on August 19, The Round 0 Market will pay on average ~~about~~ $360 per resolving claim ($4,320 total), paid proportionally by winning shares in the Round, when all answers are known (~~July  2020~~ 30 November 2020). All 12 of these claims are expected to resolve. If you have 5% of the winning shares, in any combination of claims, you earn $216.

Rounds 1 – 10:

Starting September 2019, RM will  predict replication for 3,000 claims in 4-week  rounds of about 300 claims.  In each round there will be a week of surveys followed by two weeks of markets.
Surveys are batches of 10 claims,  all from the same journal where possible. Participants must finish a survey before doing another. Participants may complete as many surveys as they want. Each batch will pay $160 to the Top 4 participants as: $80, $40, $20, $20 for a total of $4,800 in survey prizes per round. Surveys pay out at the end of each round (after markets close).
Markets on all 300 claims open shortly after the surveys close. Trading is open for two weeks, then the market is frozen until the final result is known.
We will not know which claims will resolve until DARPA reveals them to us (~~possibly not until mid~~ 2020) by 30 November 2020.
The total prize amount for a Round is the number of resolving claims on that Round (see below). That total will be divided proportionally among all winning shares. The number of shares, and therefore the associated payouts, may vary by claim.

Rounds 1-5 Markets

We expect about 50 claims to resolve from Rounds 1-5 (only direct replications count).

Markets pay about $9,000 per Round, paid on resolution, proportional to winning shares/round.
On average, that is $900 per resolving market, paid on resolution, total about $45,000 for these five rounds.

Rounds 6-10 Markets *

We expect about 125 claims to resolve from Rounds 6-10 (any replication counts). These Markets pay more per round and less per claim:

About $18,750 per Round, assuming 25 resolving claims.
Actual Round prize pool is $750 x # resolving claims.
Project reserves are used to boost per-claim $ following cut required by SCORE’s change in definition of “replication.” (Read more about this change in our blog.)

* Explanation of Payouts for Rounds 6-10 amended on 2/10/2020; updated 10 July 2020.

Return to FAQ

Want to receive updates about Replication Markets? Share your contact information below.

Explanation of Payouts

Round 0:

Rounds 1 – 10:

Rounds 1-5 Markets

Rounds 6-10 Markets *

Ready to join the project?

We’re sorry to see you go! Please visit our social media sites.