One of our forecasters wrote in with questions, and we figured that others would appreciate the answers.
Answer: Of course we want you to report the truth – near 100% here. For markets that’s clearly the thing to do, at least up to diminishing returns. But the surveys are scored using peer prediction, and some people won’t know it’s published. Should you hedge towards the mean? Not as much as you’d think.
(1) SSR isn’t a simple mean. If a bunch of people are near 100%, SSR will notice and give those coordinated estimates more weight, and judge you accordingly.
(2) It’s easy to coordinate near truth, and notoriously hard to coordinate on the elusive “beauty contest” equilibrium, even for the simple mean. Personally, I might back off slightly, but not more than 10%. I look forward to testing how that would have done.
Answer: As answered on reddit, citation counts are the sum of paper and all versions of pre-print. At least as well as Google Scholar can track them.
Answer: Do letters to the editor count?: Probably not. Letters simply announcing the preprint (e.g. Nature Briefs) don’t count. Substantial multipage “letters” may – we’re checking with experts in the field. Also, if no real peer review is involved, it’s just another preprint.
There is NO COST to participate in Replication Markets.
This research is supported (in part) by the Fetzer Franklin Fund of the John E. Fetzer Memorial Trust. It uses a platform developed for DARPA SCORE, and some staff are supported by SCORE while working on this. We are grateful for their support.
