Peer-Review vs. Info Prizes and Markets

@CB markets don’t decide anything, they simply let the crowdsourced wisdom be publicly known and provide a mechanism for inaccuracies to be instantly corrected via the price. If you’ve never had experience with ideas futures or information markets of any sort, you should really try it. Ideosphere and Predictify are a couple of the free ones, but I definitely recommend the real money versions as they are even more accurate.

I’m curious about the specific issue that brought this up. Rafe, is your position that there is any reasonable probability that there _is_ a causal link from vaccines to autism?

Our recent economic collapse should caution us against relying too heavily on market wisdom. The brightest minds in the business put millions of dollars into securities backed by mortgages that clearly could not be paid off.

I’m also wondering how a market-driven would apply to my field: mathematics. For instance, I have a paper whose results are all technically correct, but is being rejected by journals because it doesn’t fit neatly into any recognized mathematical research program. How could a market-driven system help me?

@kev, my quibble with autism/vaccine studies was posted here. Happy to discuss that, but let’s take it to that page and leave this one for peer-review v. markets.

@plektix I think the market lessons of the financial crisis are as follows: (1) more transparency in markets is better; (2) lots of leverage is destabilizing. Info markets are not inherently leveraged nor do I see a need for them to be. And they are the epitome of transparency as there is no such thing as “insider information” when it comes to truth claims, in fact the market is *designed* to out the truth at every turn.

As for math apps, look on Ideosphere for claims such as P!=NP and Goldbach’s Conjecture. More specifically regarding research, read Hanson’s papers comparing how peer-review works and how information prizes (and info markets) work when it comes to academic research. The papers address exactly your problem.

Firstly, I would like to thank Rafe for this medium to discuss our differences of opinion. The 140 character limit was becoming a little too restrictive for both of us to adequately express our positions.

I should begin by stating that my position is not that peer-review doesn’t have inherent flaws. It’s a human enterprise, and as such, is prone to human problems. However, I believe that peer-review is the most effective and practical system we have for maintaining high scientific standards.

Do I think that info markets work? Sure, but in the same sense that Wall Street and betting shops work. Do I think that info markets have a place in science? No.

First, you need to have an understanding of what constitutes science. Science is a largely blind process, in the sense that we don’t know the results of an experiment a priori. Otherwise, we wouldn’t need to do science. In this sense scientific enquiry and info markets can be said to be similar. But the similarities stop there.

In science we can never be said to know the true answer to any question. The philosopher David Hume first pointed out the problem with induction in the 18th century. In short, past experiences cannot be used with certainty to predict future events. His ideas were later formulated, mostly by Karl Popper, into the falsification principle, which forms the backbone of modern science. Richard Feynman said it best concerning scientific enquiry: “we never are right; we can only be sure we’re wrong.” Info markets appear to dependent on post hoc knowledge of the outcome of a particular event or parameter. In contrast, all scientific results are tentative.

In order to bring this discussion forward, I suggest Rafe present me with a concrete example of how info markets could be applied to actual scientific research.

Firstly, I would like to thank Rafe for this medium to discuss our differences of opinion. The 140 character limit was becoming a little too restrictive for both of us to adequately express our positions.

I should begin by stating that my position is not that peer-review doesn’t have inherent flaws. It’s a human enterprise, and as such, is prone to human problems. However, I believe that peer-review is the most effective and practical system we have for maintaining high scientific standards.

Do I think that info markets work? Sure, but in the same sense that Wall Street and betting shops work. Do I think that info markets have a place in science? No.

First, you need to have an understanding of what constitutes science. Science is a largely blind process, in the sense that we don’t know the results of an experiment a priori. Otherwise, we wouldn’t need to do science. In this sense scientific enquiry and info markets can be said to be similar. But the similarities stop there.

In science we can never be said to know the true answer to any question. The philosopher David Hume first pointed out the problem with induction in the 18th century. In short, past experiences cannot be used with certainty to predict future events. His ideas were later formulated, mostly by Karl Popper, into the falsification principle, which forms the backbone of modern science. Richard Feynman said it best concerning scientific enquiry: “we never are right; we can only be sure we’re wrong.” Info markets appear to dependent on post hoc knowledge of the outcome of a particular event or parameter. In contrast, all scientific results are tentative.

In order to bring this discussion forward, I suggest Rafe present me with a concrete example of how info markets could be applied to actual scientific research.

Rafe, I was asking because I thought we could come up with an illustrative bet or assertion. But as your quibble is with a particular study, a results-based bet is not as well motivated.

My personal opinion is that the best approach is a mix of publication and markets. For what I’d call “exploratory” science, peer review is probably better. You’re trying to generate novel explanations.

For what I’d call “confirmatory” science, markets are probably better. You’re trying to evaluate the relative merits of alternative, well-defined hypotheses (or apply a bunch of hypotheses to solving a particular practical problem).

@Ciaran. I have a good example of how to apply markets to science: global warming.

We have two competing hypotheses. One says that CO2 emissions will warm the Earth by more than 2.5 deg C in the next 50 years. One says CO that CO2 emissions will warm the Earth by less than 1.5 deg C in the next 50 years. Between 1.5 and 2.5 deg C, it’s a push. (I’m simplifying things for clarity at the moment. There are actually more than two alternatives and they are a bit more complicated than just temperature assertions.)

We’ve got to make real policy decisions on this topic now. A market for contracts on future temperature would illustrate the true “consensus” much better than publishing counts.

The suggestion that info markets can replace peer review implies that peer review is an irreparably defective system. As I’ve stated before, peer review may not be a flawless system but it is far from broken. Peer review ensures scientific quality control. The refereeing standards vary from journal to journal but scientists themselves are well aware of which journal publish quality research. For instance, the New England Journal of Medicine and Nature publish between 6% and 8% of submitted material. That is a very high attrition rate. The peer review process helps academics like myself plough through the relevant literature in a more efficient manner. Rafe suggests that conflicts of interest pose serious problems for peer review. In fact, the most prestigious journal go to great lengths to actively avoid conflicts of interest from ever arising. Tighter guidelines could certainly be implemented regarding conflicts of interests but a complete overhaul of the system would be excessive.

There is an implicit presumption by the proponents of info markets that scientists aren’t being as honest as they could be. Do you honestly believe that scientists spend their time slogging over theories that they believe in their heart to be hogwash? Most scientists believe that their methods and findings are correct. When new techniques and models that better predict a phenomenon are discovered scientists are generally quick to adopt them. When scientists publish their findings in a peer reviewed journal they are demonstrating confidence in their findings. If scientists were to bet on scientific ideas they would bet on their own, almost without exception.

Info markets are predicated on financial profits and losses motivating people to make correct decisions. There are very few scientists that entered the field who were motivated by money. Believe me, there are simpler and better ways to make money than through science. Scientists are motivated primarily by their innate curiosity. Scientists are among the last group of professionals that would be attracted by betting markets and without expert opinions the proposed scheme would be little better than a lottery. Beyond the hard sell we don’t need scientists being distracted by betting. We want researchers to focus on their research, while minimising any distractions.

Kevin’s example suggests that info markets have a limited application in a single stage of the scientific method, i.e. prediction. However, in science we are more interested in the base models underlying these predictions, in other words the “why” and “how” questions. So far, I’ve seen no example of how info markets can give us insights into these types of questions. Further compounding the issue, is the fact that predictions are completely dependent on their underlying hypotheses. Since we can never be certain of the validity of our hypotheses, we can never be certain of the validity of our predictions. The truth is that info models don’t tell us much more than what scientists could easily acquire from studying the peer reviewed literature.

The use of prediction markets in regard to global warming is highly impractical. Do we sit around and wait for 2050 to hit, so we can say “I told you so”? In the real world we act on the best current information regarding issues like global warming. However, in the intervening time we will enact policies that will impact global warming and any previous predictions we had made. Moreover, the best model may not be the best predictor because of unforeseen changes or the availability of new data, while a bad model, using sloppy data, may stumble upon the correct prediction by chance alone. I am also left wondering how info markets deal with questions that are not going to be decided any time in the near future.

Info markets also fail to take into account theoretical fields of science. These are by definition exploratory and subject to fail in a great many cases. Info markets would classify a theoretical model that doesn’t predict a future event as a bad model, while science would classify it as progress, since it allows us to falsify a particular hypothesis. Are you suggesting that we discourage fields such as theoretical physics by means of financial penalties? I see info markets favouring scientists who make correct predictions, while restricting originality and exploratory work.

The suggestion that info markets can replace peer review implies that peer review is an irreparably defective system. As I’ve stated before, peer review may not be a flawless system but it is far from broken. Peer review ensures scientific quality control. The refereeing standards vary from journal to journal but scientists themselves are well aware of which journal publish quality research. For instance, the New England Journal of Medicine and Nature publish between 6% and 8% of submitted material. That is a very high attrition rate. The peer review process helps academics like myself plough through the relevant literature in a more efficient manner. Rafe suggests that conflicts of interest pose serious problems for peer review. In fact, the most prestigious journal go to great lengths to actively avoid conflicts of interest from ever arising. Tighter guidelines could certainly be implemented regarding conflicts of interests but a complete overhaul of the system would be excessive.

There is an implicit presumption by the proponents of info markets that scientists aren’t being as honest as they could be. Do you honestly believe that scientists spend their time slogging over theories that they believe in their heart to be hogwash? Most scientists believe that their methods and findings are correct. When new techniques and models that better predict a phenomenon are discovered scientists are generally quick to adopt them. When scientists publish their findings in a peer reviewed journal they are demonstrating confidence in their findings. If scientists were to bet on scientific ideas they would bet on their own, almost without exception.

Info markets are predicated on financial profits and losses motivating people to make correct decisions. There are very few scientists that entered the field who were motivated by money. Believe me, there are simpler and better ways to make money than through science. Scientists are motivated primarily by their innate curiosity. Scientists are among the last group of professionals that would be attracted by betting markets and without expert opinions the proposed scheme would be little better than a lottery. Beyond the hard sell we don

@kevin, one of the main problems with peer review is that it stifles novelty. If you are a reviewer are you going to give more credence to something you know a lot about or something that is new and perhaps antithetical to what you believe?

I’m in favor of publishing results. I just think that instead of peer review we should follow a PloS One model and attach at least two types of markets to each publication (a) truth value (b) importance/relevance.

And I think info prizes can be used to change the bad incentive scheme that has emerged where scientists are forced to “publish or perish”. This leads to micro-focused (i.e. irrelevant) incremental results that you carve out and are chosen so as to be so arcane that nobody could ever challenge your work.

Peer-review has a place in science, just not the dominant one. And it should always be checked by markets.

@rafefurst: How do we determine the truth value of scientific research?

@rafefurst: How do we determine the truth value of scientific research?

@Ciaran

Here is a simple example.

Will Omega-3 supplementation improve mental performance?

I am confident that info-markets will more accurately and more efficiently (faster and cheaper) give us the answer to this question than any traditional research methods involving peer review.

@Ciaran. Prices today incorporate expectations for the future. The price of pork belly futures today incorporates everyone’s expectations about events that will effect pork belly supply and demand in the future.

Similarly, the price of contracts today on different predictions about temperature in 50 years incorporate all of the expectations about how well current models correspond to reality. Yes, these contracts don’t pay out for 50 years. But we can use today’s prices to act. That’s the whole point.

@Rafe. My statements about the usefulness of peer review were only in contrast to prediction markets and only for a very broad definition of peer review. I include PLoS ONE under that big tent.

@daniel et al

People buy homeopathic remedies to the tune of $150 million a year. Is the market saying it works?

Science says homeopathic remedies don’t do what they are purported to. Overwhelmingly.

The masses can be asses and therefore markets can be woefully misleading. It’s hard to blame the markets; they are, afterall, just markets not sentient beings, and as such just reflect the collective input of the constituent market participants. But garbage in, garbage out.

Peer review is useful to filter out garbage.

“There is an implicit presumption by the proponents of info markets that scientists aren’t being as honest as they could be. Do you honestly believe that scientists spend their time slogging over theories that they believe in their heart to be hogwash?”

To be clear, I was not referring to the researchers but rather the reviewers. And I wasn’t suggesting that it is a conscious bias, but rather an unconscious one. For instance if I asked you who will win cricket match this weekend, aren’t you inclined to your home team more frequently than can really be justified by the stats? But now if I asked you to bet $1000 on the outcome of the game, I am pretty sure you will forget your wishful thinking and bet with your clear, rational mind. Research bears this out that time an again when there’s money on the line, people are more rational, better predictors, clearer truth-seers than when there’s not. And info markets are a way to leverage this phenomena to the advantage of all those who seek truth.

@torsten, actually yes, the market is saying that homeopathy works. But it includes the placebo effect, which we know is upwards of 40% of any medical intervention.

We should be debunking the crap that pharmaceuticals put out that shows zero effect once you control for placebo (like cough syrup). “Medicine” is typically way more costly than homeopathy for the same outcome.

Rafe, you can throw in about half of all medical spending into the placebo bin:

http://www.cato-unbound.org/2007/09/10/robin-hanson/cut-medicine-in-half/

http://hanson.gmu.edu/showcare.pdf