The truth is out there

May 20th, 2010 by Tommaso De Benetti

The amount of information available today is vast beyond our comprehension. The very discussion of this data deluge has itself generated a fair bit of coverage. Still, like the elephant in the room, it is not often that you hear people complain about the fact that, let’s face it, most of it is garbage.

It might sound a bit catastrophic, but the numbers look scary: information overload could be the next weapon of mass instruction. Some facts about this exploding data: YouTube turns five years old this month, with an average of more than two billion views per day. At the beginning of 2010, Technorati alone featured 133m blogs. There have been over 14.2 billion tweets, while over five percent of the world’s population now use Facebook. Across all these forums people can produce content with no authority, entitlement or credibility.

Believe it or not
Let’s be clear: social media and web tools exist to make it easy for people to craft and share amusing content. Of course, this ease also allows the creation of huge amounts of imprecise, irrelevant or deliberately deceitful information. Whether it’s reports that downplay global warming, give prominence to an orphanage for sloths on the front page of a major newspaper, or a rant on how a corrupt government is replacing religious morality with game theory, the problem is the same: how can we filter out the garbage from the quality content, thereby allowing us to develop original, well informed opinions?

A Denver-based startup called HowTru claims to have the solution. Whenever a HowTru user reads a story, he or she can either confirm or refute its validity. In time, the story will develop a veracity score visible to HowTru users indicating how reliable a specific source is.

This sounds great in theory, but I wonder how well it will work in practice. At best, it seems the majority of HowTru users will base their judgments on information absorbed elsewhere. Given the dire shape of worldwide journalism, even opinions based on supposedly reliable channels are likely to be biased or inaccurate. And that’s at best. At worst they will be uninformed opinions based on the whims or worldview of the user. This raises the question: can we trust the wisdom of the crowd?

Too good to be true?
Toma Bedolla, founder of the company, claimed in an interview with ReadWriteWeb that: “As we reach critical mass, we can counter the effects of misinformation.” To me this is like saying because a Britney Spears album sold millions of copies worldwide, it is automatically worth listening to.

Apparently the system has some countermeasures for this problem. The website states “Accuracy of the article is verified or refuted by members of the crowd. These evaluations are not counted equally; HowTru weighs each evaluation according to the credibility earned by the evaluator in the relevant context, e.g., nutrition, business, politics etc.”

Again, this sounds good, but I still think it is problematic. Truth is not a matter of democracy or consensual majority: the Pythagorean Theorem would hold true even if not a single person in the world believed in it.

Comments under the article demonstrate similar skepticism. As one user argues “there was a time when the crowd believed the Earth was flat.” Unfortunately, popularity is not necessarily a reliable indicator when evaluating if something is true or not, especially because, as another user claims “People gravitate towards their tribe. It’s false precision”.

In response to such arguments one of the developers responds: “We make considerable efforts to filter out the effects of ‘popularity’ on credibility (real and perceived). By doing so, we create a system that provides diminishing returns for someone who follows the crowd.”

Even so, I suspect that Howtru ratings will still reflect whatever the predominant bias of the particular audience is, regardless of what the underlying facts may be (eg the same story posted on Fox will have a different rating than if it was published on The Economist). This is not to say that the users are disingenuous – rather that it is human nature to interpret information in a way that reflects one’s pre-conceived ideas – even if it means ignoring the elephant on your coffee table.

Share Print article

Pingback: Tweets that mention The truth is out there -- Topsy.com
Petri

Aa, should have known it was an elephant! Thought it was a mouse when the cabinet meeting started.
Sami Sundell

This story highlights the problem many crowd-based efforts have: “wisdom of the crowds” is an oxymoron that promotes democracy over facts.

Consider HowTru in Finnish press: Seitsemän päivää is the most read magazine in Finland, in newspaper category Ilta-Sanomat and Iltalehti are second to only Helsingin Sanomat. Journalistic value of the yellow press in general is highly doubtful but people still read them. Is it really likely that people who read them will vote them down as crappy journalism?

Now that there's more and more web sites focusing on gathering information using crowds, it also highlights another problem: it's enough to get facts wrong in one prominent place, “wisdom of the crowds” will then multiply it and it becomes The Truth. Once misinformation has propagated through network, it's again so much harder to rectify it.

Wisdom of the crowds requires there are actually knowledgeable and insightful individuals forming the crowd, or that the task at hand is simplified so that even the weakest link in the crowd is likely to make the “correct” choice.
http://www.parliamodivideogiochi.it Tommaso De Benetti

My feeling is that we are not getting away with opinion leaders any time soon.

If the 12-13-14 of September 2001 you would have asked an average citizen of Texas if Middle-East should be wiped out in its entirety they would have very likely answered “yes”. Crowds are emotional, often not enough informed on the facts or informed only by a limited range of sources. Very few people spend enough time trying to understand facts from all the viewpoints and all their implications. Even history tends to be written (or rewritten) only from a specific perspective.

In my opinion, giving the power to decide what is true to the crowd makes truth dependent on a “democratic trend”, that has very little to do with truth itself.
http://twitter.com/jussir Jussi Rytkönen

Wisdom of crowd is one thing, but much bigger problem for HowTru will be to reach the critical mass. There's no way people are going to use their current interface (click the javascript link on random web pages), whitout getting frustrated after few “no reviews, please wite one” responses. Nice idea, though.
http://twitter.com/jussir Jussi Rytkönen

I think you'r missing the point of “wisdom of crowds” (as presented in the book with the same title). For crowd decision to work it requires the right conditions, and taking something like the popularity of newspapers is hardly a good mesure, since there are so many things that can affect that. But when the conditions are right the crowds do tend to outperform experts almost always. One of those required conditions, by the way, is independence of decision agents (to avoid group thinking).
Sami Sundell

Yeah, I was more referring to the general notion of “gather the crowds and something good will happen” which has been something of a trend lately, without too much of a look at what kind of people form the crowd.

Popularity of newspapers relates to particular case of HowTru in the sense that they are probably hoping to get everyone to use it, and “everyone”, on the average, seems to be reading Seitsemän päivää. In other words, I'm a bit sceptical on how those requirements are fulfilled in this case (or in many other cases which rely on generic Internet populace participating), when the crowd quality assessment is applied to journalism in general.

Don't get me wrong, I think that kind of system would definitely be a service to mankind, and I would definitely hope for HowTru to succeed. If they manage to keep that group thinking and popularity contest aspect out of the picture, I'm doubly impressed.
http://twitter.com/rikkomba Da Livin' Rikkomba

laimed in an interview with ReadWriteWeb that

that line creates an error in the text. it's not working, like that.
http://about.me/toma.bedolla Toma Bedolla

Just “stumbled” upon this blog post…Tommaso, I thank you for taking notice. In reading some of your comments here, it is in line with the usual skepticism that we receive. There are a couple of things that people don’t quite “get” when we talk about solving the problem of misinformationnn1. In any single instance, the wisdom of the crowd is very dependent on the “right conditions”. What we’re doing is creating an aggregation of the history of everyone’s assessments within relevant contexts to determine just how “credible” each member of the crowd is within the context of any article they evaluate. It takes time to vet any one piece of information, but we can begin to learn the credibility of individuals who participate and weight each vote accordingly. This means that one person can counter the popular assessments of a much larger group.nn2. Evaluations or votes are dynamic, which means that as one’s credibility evolves, the weight of their past evaluations also change. An evaluation that helped yield a high veracity score today may end up hurting the veracity of that article tomorrownn3. Which brings us to our third point which is that information should follow you after you’ve consumed it. This means that I read your blog today and it has an accuracy score of say 73.4%. Some new information comes out and it starts to fall. Let’s say I have asked to be notified by HowTru when the VeracityScore of any information shifts 50% or more. Score hits 20.5% and I get an e-mail that says, “The article you read/evaluated “Title of Article” has fallen to 20.5%; click here to find out why”. This way information that you think you know is constantly refreshed appropriately.nnThis is just some of the ideas that we’re working on at HowTru. We look forward to turning everyone’s skepticism into support.nn-Toma BedollannFounder/CEO Veracious Entropy and the HowTru? project.
http://www.parliamodivideogiochi.it Tommaso De Benetti

Hello Toma, and thanks for clarifying the system. It looks like the biggest challenge for HowTru is to communicate in just a few words how it all works. When we talk about “defining the truth”, it’s easy for people to become radical all of a sudden
http://about.me/toma.bedolla Toma Bedolla

You couldn’t be more right ;^)nnWe’ve been working hard on the language of it…HowTru is our second name. It was originally called Inforesting (Information + Foresting). Apparently we were too clever for our own good. Thanks for taking notice…we hope you’ll help spread the word when we launch our alpha in the coming weeks!nn-Toma
M J Horn

Tommaso – how refreshingly true! This is a spot-on, on point article. The opinionated, assumption laden crowd has become the abritator of concensus that cannot be relied upon. Social media whilst great in lots of respects to allow people to be critics, do and say what they want in a free society, but have none the less been turned into the measure of public opinion. Most of the opinions they hold are based on very little understanding of the subject upon which they are commenting, Its off the cuff opinion based on scant research, no consideration of both sides of the arguement and only considered as to how it affects them personally (or how thjey think it does, without looking any deeper)

That makes crowd opinion a very poor basis for reasoned and logical judgement that past unbiased journalism could previously be relied upon to inform or at least communicate a fair assessment based on some level of factual research.

Thanks for the article