Resolutions for Replication Markets!

When Replication Markets launched in 2019, we could not have expected the global interruptions that affected our project timeline. Finally, we are ready to resolve and pay out our markets! One silver lining: in waiting to announce results, we have more resolutions to pay out.

More resolutions means a larger sample size to answer the meta-questions of Round 0.

Our patience has produced more resolutions from claims in Rounds 1-10. RM’s users traded on 77 of the 79 claims that qualified for payout.

We made the best we could of the pandemic by adding Round 11 for preprints of COVID-related papers. This has paid off — both for our research and for RM’s traders: 30 of the 100 claims from Round 11 have resolved.

While RM knows the results of the replications, we are not allowed to share the individual findings. We can share the information in aggregate, which we present in the tables below. Click on a link to jump to the section.

Resolutions per Round

Leaderboard for users and their prizes

Round 0 Meta-Questions

Resolutions per Round

Note that Claims is the number of eligible resolved claims that users traded on.

Round Claims 
118
214
37
47
55
69
79
84
92
103
11

30

Return to top

Leaderboard for users and their prizes

  • Winning users should look for an email from domain @replicationmarkets.com by Wednesday, November 17.
  • Make sure to add our domain to your allow-list so the email lands in your in-box!
username prize total
BradleyJBaker $5,721
meaning.mosaic.curtain $4,229
unipedal $4,020
mVranka $3,903
physwiz $3,615
mbulutay $3,172
sattuma $2,827
ejorgenson $2,822
CPM $2,692
Nokta $2,610
namk $2,572
jc20 $2,557
JonS $2,303
Scalene1234 $2,290
RobC $2,226
stanlaurel $2,119
galbuquerque $2,055
DavidRhysBernard $1,974
levent $1,841
Taku $1,789
Vitorper $1,533
Luthor113 $1,483
Matho $1,425
martin_kosiik $1,414
AchinToBe $1,413
gordon gekko $1,364
andi $1,353
GeritPfuhl $1,333
guilfoos $1,188
ted $1,175
CesarMant $1,172
curious $1,149
yetAnotherTrader $1,148
Mptg2 $1,143
acct $1,137
DMS $1,105
ncerutti $1,090
tanzor $1,063
JozefBavolar $1,060
maalsc $1,045
wetpaint $1,027
Sceptic $1,017
Sjaakharrie $983
denns $956
kushkan $944
sflicht $934
Poison $925
Franzi $919
pbog $903
antipodeanlurker $894
shkaff $889
felixbunny $878
Miss_prediction $875
friendlysnail $875
zultan $839
NunoNunez $809
adambear91 $781
relihan $772
Bitschnau $750
Pavol $742
Hygglot $738
BBalmford $735
cgravert $731
benthamite $714
rm_user $713
cmeinel $704
bjaeger $693
Frantisek $664
smingers $653
NLA01 $653
Cory $638
educatedguess $630
xoph09 $626
jonathanstray $608
Econ101 $599
EirikStroemland $561
Miranda $560
Troof_ $559
ekarpinski $548
jacobjacob $540
dschwarz $533
Ilei $533
Pepe $532
smeinz $504
e4e6 $497
Hmassz $489
buddhabrot $481
bmandl $474
lincoln $468
drl1764 $467
chiefi_lux $455
thekalby $454
catrachohansen $447
prorgan $443
smetis $443
Kirstin $439
Marol198 $425
Merkus90 $421
gingervitiz $406
dhitch $399
Sclamons $398
ClaudiusMaier $391
riarm $370
braoi $363
kongk $363
Illmatic $352
lukasneville $352
Reviewer3 $345
cwilliams $343
renebekkers $343
noob_trader $336
mspape $325
vaquinha $315
ulrich $312
Lenkafiala $307
Botilicious $299
radiator $292
wangtong1235813 $289
bgfarrar $268
jpheffne $264
paul_henne $258
unpredictified $256
WhatCouldHappen $250
MariaC $244
mg $240
Humboldt $238
pwaffle $238
cavanbonner $237
yeli23 $236
abaillon $229
wellfleetOyster $220
GKarreskog $216
sutych $213
bjoerksb $212
felize2000 $212
nurit.nobel $210
andrewsmith $193
philip_p $192
KABrekke $191
adamlgreen $188
nicolenightingale $186
NPXY $185
Sampotarmo $182
vtevelde $179
PTan $177
rboehm $174
Dima $168
jaynbaxter $167
benkenobi $167
robert.thibault $166
scchen $158
Poleres $149
John Prowse $148
DCH $147
jornstein $145
SBNDad $144
mds $143
hugopedder $142
FloraCozzi $141
nesealkan $141
schniter $138
cbrandolino $137
Brett $135
mf483 $134
SMGifford $130
alexrjl $127
Matti Heino $127
mpaaso $127
insatiableurge $126
SeaSantiago $124
nathan98000 $119
dankee_yoodle $119
mvernholmen $115
Pheck100 $113
8435207df859e854 $112
Garonmae $112
mattathias $104
ocean01234 $102
jjxw $101
SarahSchoch $100
caroroux $98
shig_ono $96
pboenisch $96
Davidoj $95
Machery $95
Nimchuk $94
simoncolumbus $94
spahen $89
An1lam $89
Bvschuer $89
HWillem $88
warsquid $86
Laura $86
gracecsmith $83
Jo.Weiner $83
slantedlight80 $80
JCmo $75
HABK $75
ytan830 $74
andrasm $73
enr1co $73
wscampbell $70
rpeeters $68
Tamay $67
bravodelta $67
CHartling $63
sharedvariance $60
bphillab $59
dstauffer11 $58
2e10e122 $56
Aniaw81 $54
jbhender $53
Spy $52
opowell $47
sarah.jacobson $46
Azaza $46
ecotico $46
c_cap $46
chewy $45
grpennycook $42
lohsej $42
boris $41
cmadan $40
pthiemann $38
matthias $38
mvs $37
Danva85 $37
Danielf089 $35
gstupp $34
adamwhlr $33
funashi $32
j19 $32
joshfoster $32
nwblack $31
m.stroom $31
ajaxecles $31
ekimbrough $26
brungl $26
TomL $25
oneAtATime $25
TDZ $24
drichter $23
tankred $23
malsh $23
JohnH $23
Zubrin $23
AMW $23
alf847 $22
OliverGenschow $21
baxpr $20
bshor $20
paulalab $19
jwulff $19
cflexman $19
Wolfschi $19
Vlad $17
ytmytm $17
benthinkin $16

Round 0 Meta-questions

  1. ‘What will be the average replication rate in score?’
  2. ‘What will be the average replication rate in economics?’
  3. ‘What will be the average replication rate in political sciences?’
  4. ‘What will be the average replication rate in psychology?’
  5. ‘What will be the average replication rate in education research?’
  6. ‘What will be the average replication rate in sociology and criminology?’
  7. ‘What will be the average replication rate in marketing, management and related areas?’
  8. ‘What will be the average replication rate for papers published in 2009/10?’
  9. ‘What will be the average replication rate for papers published in 2011/12?’
  10. ‘What will be the average replication rate for papers published in 2013/14?’
  11. ‘What will be the average replication rate for papers published in 2015/16?’
  12. ‘What will be the average replication rate for papers published in 2017/18?’

Return to top

2 thoughts on “Resolutions for Replication Markets!”

  • Ted says:

    Awesome. Would be interesting to see (a) the aggregate replication rates (with error bars) answering the round 0 questions as well as (b) the trading volumes of each of the winners or similar accuracy metrics. Curious to gauge how much of the difference in payout is due to more trades made vs better judgment.

    • Charles Twardy says:

      Definitely interesting – just trying to figure out how much we can release without violating SCORE embargo.

      We are revealing the overall replication rate (so far) was 54% — using all 100 of the non-covid resolutions we have received. (Uses both direct and data analytic replications for a better estimate.) That’s unofficial but the the sample is large enough it should be pretty stable and non-identifying.

      The per-field rates varied quite a bit, but some samples are small (<10), so we're holding those until we can confer more with COS.

Leave a Reply to Ted Cancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

May we send you invitations to future research projects? 

Add your email address to our low-volume distribution list.

Note: We cannot re-use participant lists for new recruiting. Please opt-in (above) for occasional announcements about related future studies.

We’re sorry to see you go! Please visit our social media sites.

This site uses cookies to provide you with a better browsing experience.

Visit our Privacy Policy for more information.