Who will police The Police?

February 23rd, 2010 by

Police appearing at the Xmas party by wili_hybrid

A while back I had a bit of trouble with The Police. It’s not a period of my life I am proud of. Despite it happening years ago, I am only now able to talk about it.

The incident occurred when The Police were involved in the release of the song “Every Breath You Take”. In it, the lead singer Sting opines “How my poor heart aches”. I misheard the lyric as “I’m a pool hall ace,” often repeating the catchy lyric ironically when missing shots in games of pool. No one ever told me of my mistake. I guess either my friends thought I was joking or they were amongst the many others who made the same mistake. Even so, the memory still haunts me to this day.

But despite the evidence this incident provides to the contrary, I still like to think that my understanding of English – even when mumbled – would make me a passable translator. As discussed in the earlier posting by Tommaso on this site, crowdsourcing of translations now gives me that opportunity. Today, with help from enthusiasts around the world, outfits like Italian Subs Addicted provide fast, accurate subtitling of English films and television.

One of the many interesting aspects of their system, especially for a Pool Hall Ace like me, is how it deals with quality control.

Peering Over Your Shoulder
As explained in the earlier blog, Italian Sub Addicted chops programs into small pieces and farms them out to groups to translate. To maintain quality, once completed, each of these pieces is peer reviewed by the other groups involved.

They are not the only ones making use of this peer review system. A couple of years back, Facebook asked users to translate the site from its original English format to other languages. Keen to have Facebook available in their own language, amateur translators signed up in their thousands and did it for free.

Quality control in this case was provided in the form of a voting system that weeded out bad translations. Due to the enormous number of people willing to participate in the voting, this proved highly effective. In fact, Facebook has such faith in the system that has since begun to offer this service to other websites.

Into The Unknown
But even Face-lators (catchy huh?) in their collective thousands may have had a problem deciphering this speech by Donald Rumsfeld. In it he mentioned “known knowns”, “known unknowns” and “unknown unknowns” and confused the hell out of a lot of people – regardless of whether English was their mother tongue.

I can’t say for sure, but my guess is that our friends at CrowdFlower probably would have got the gist of it. As providers of crowdsourced labor to institutions like Microsoft and Princeton University, they make it their business to provide extremely high quality output using a quality control system called CrowdControl.

CrowdControl works by using Rumsfeld’s known knowns to evaluate just how good people are at identifying known unknowns, if you know what I mean. When they provide microtasks for someone to complete, they include some tasks they already know the response to. If a worker errs on one of these tasks, they are notified, in the expectation that they will learn from the mistake. It also allows CrowdFlower to identify over time who gives results of consistently high quality. The website doesn’t say how continual underachievers get dealt with in the long run; perhaps a notification that lets them down easy: “Things just aren’t working out. Your skills would be better suited to something requiring less intelligence.”

Amazon’s Mechanical Turk won’t even let life’s underachievers get that far. This site exercises quality control through many of their microtasks, or Human Intelligence Tasks as they call them, requiring workers to earn certain qualifications before they can attempt them. Anyone can log on and try to earn these qualifications – and it’s worth doing the study. A quick scan of the site shows that these qualifications are a prerequisite for many of the highest paying microtasks. So stay in school kids!

This is not to say that all crowdsourcing tasks require qualifications, or even the most basic level of intelligence. Some tasks are as simple as indicating whether a person in a photo is a man or a woman. (Having said this, gender differentiation in today’s world is often far from easy).

There are an ever increasing number of microtasks available on the net. From this, we can infer that the quality control systems in place must be working effectively. This is because, regardless of how cheap crowdsourcing microtasks is, companies would not continue to shell out money if they weren’t getting accurate information in return.

In hindsight, I could’ve used some of that quality control around the time of my misdemeanor. A little bit of peer review on my friends’ parts wouldn‘t have gone astray. In fact, it would have saved me a lot of embarrassment if they had taken it upon themselves to police The Police.