Peer-Review vs. Info Prizes and Markets
I have been having a 140 character discussion with Ciarán Brewster (@macbruski) via twitter. And while it’s kind of interesting to force complex subject matter into very few characters, it is limiting the discussion, so I will summarize it so far here and hopefully others can weigh in too.
Ciarán Brewster: Peer-reviewed studies that have looked at the possible relationship between vaccines and autism: http://bit.ly/BOIcw
Rafe Furst: Peer-reviewed autism studies can produce bad science too http://tinyurl.com/d9dpde
CB: If peer review is such a bad system, I’d love to know what alternative you suggest. Please enlighten me.
CB: Hanson doesn’t address how we’re supposed to validate results, especially in a system where researchers are driven by money.
RF: you could use something like ideas futures or truthmarkets http://tinyurl.com/dchxlo
CB: It seems that truthmarkets make the big assumption that we have a way to evaluate the likely validity of a claim
CB: What’s to stop a group of scientists, driven by money, propagating a lie, since the patron has no access to relevant info?
RF: markets r open 2 every1 so there is incentive 4 any1 with more truthful info 2 make a buck. Suspect fraud? Redo expmt & bet on it
CB: Redo expmt & bet on it. But this still doesn’t solve how we determine the validity of the suspected or new claim. Who decides?
RF: the market decides; it’s consensus just as with all “truths” but with their money on the line people are more accurate/honest
CB: So it’s just money driven peer-review then. You seem to imply that scientists are dishonest unless there’s money on the line.
RF: 1. Thousands of peers betting beats 5 with inherent COI 2. Humans r better predictors & assesors of truth when own money on line
CB: “1000’s of peers betting.” I’m sorry, but what universe are you living? People will do and say anything if price is high enough.
RF: “People will do & say anything if price is high enough”. And other people will sell, driving price down & make a profit off them
CB: Could you please give me some examples of testable scientific hypotheses using the model of info markets?
RF: markets can’t set up experiments, test hypotheses, or get results; they can provide incentive for humans to do it very well
CB: If info markets are the ultimate arbiter, how do they decide what is valid? How do they have knowledge of quality science?