Crowdsourcing Truthiness

Google Labs has a new service called “In Quotes” which might be tweakable to do a truth market of sorts.  Here’s the suggestion I just emailed them on this topic:

I would love to use Google “In Quotes” to crowdsource measures of truth.

For instance, I just saw this:

“In a world of hostile and unstable suppliers of oil, this nation will achieve strategic independence by 2025,” said Mr. McCain during a campaign speech. [ Wed, 29 Oct 2008 Washington Times ]

I would like to be able to indicate on a scale from 0 (false) to 10 (true) whether I believed what McCain said is true (that we will achieve strategic independence by 2025).  Everyone’s rating would yield an average number (let’s say it was 6.8).  In addition, all other quotes that were attributed to McCain would have a truth index too, and the average of those numbers would be his dynamically updated “truthiness” rating.  You could add a decay factor to allow for people’s reputations to change over time.

You could then sort quotes and people by their truthiness rating right along side date and relevance when people do searches on news items.

Additionally, you could get a truthiness rating for the sources by averaging the truth index of each quote they publish.  Some media like to quote trolls so as to trump up gratuitous controversy, and this would separate those sources from the true investigative journalists.

I can think of many problems with this approach, but I’m curious.  What do you think is the most likely way this could fail to achieve the objective of measuring truth?

  • Aside from the fact that “truthiness” is a word coined by satirist Stephen Colbert to mean (paraphrasing) ‘what feels true in your gut, not what some books might say’, what’s really being measured here is more like “believability”.

    What you’re going to get is a combination of
    a) people who like John McCain
    (popularity of speaker)
    b) people who think John McCain believes what he says
    (honesty)
    c) people who think what John McCain says is correct
    (accuracy)
    d) people who think what John McCain says is desirable
    (popularity of proposal)
    but in this case, NONE of these people know the correct answer, because it’s a prediction, not a currently verifiable fact.
    Also, if there is some uncertainty in the quote, does this make the quote less true or more true?
    You really need more than one dimension to quantify it (accuracy vs certainty etc).

    All that being said, this might produce some interesting data, I’m just not sure how strongly correlated it would be with objective reality.

  • Alex Golubev

    Definitely not the same, but related – Book of Odds launches on 10/14/09:
    http://www.bookofodds.com/