The secrets of Digitalkoot: Lessons learned crowdsourcing data entry to 50,000 people (for free)

June 16th, 2011 by

microtask_digitalkoot_wrapupAs many of you will know, a few months ago Microtask and the National Library of Finland launched a project called Digitalkoot. Being the first Microtask-powered public service, Digitalkoot was a huge test for our crowdsourcing platform, the use of volunteers to complete microtasks, our ideas about gamification and rewards, and basically us as decent, upstanding citizens.

As we have already mentioned, with over 50,000 volunteers so far, the project has been a great success (which is nice for us, because it means we can start sleeping at night again). We have learned so much from this experience that we decided to share it with you, the crowd that made it all possible.

Making work play: Move over Super Mario, hello Super a-Mole-d
For those of you who have not understood a word I have so far said, the aim of Digitalkoot is to accurately digitize the National Library’s enormous archives, making them searchable over the internet. It uses crowdsourced volunteers to input data that Optical Character Recognition (OCR) software struggles with (for example documents that are handwritten or printed in old fonts, such as very old copies of the newspaper Aamulehti).

Digitalkoot relies on machines, humans and a gaming twist to make it all fun. This is how it works in practice: old text newspaper is scanned by OCR software and then cut up into individual words. These words are sent to volunteers in the form of two online games we created (Mole Hunt and Mole Bridge). Volunteers must accurately decipher the words in order to achieve certain game objectives, such as helping the moles cross a bridge or keeping them away from a garden.

What for volunteers was an amusing game, for the National Library was an affordable way to digitize, bit by bit, the entire collection of Aamulehti. This is what The New York Times, Wired and others said about our heroic, text-loving moles.

Engineering success: Golden tasks and diligent pests
The challenges Microtask’s engineers had to overcome were daunting (we will present a paper on Digitalkoot at HCOMP 2011). One problem, for example, was how to deal with malicious players who deliberately type in the wrong words (one tireless volunteer spent over an hour and a half doing just this. We like to think he/she is either a very bad typist or has a passionate hatred for moles…either that or someone let their dog volunteer).

To identify such volunteers, the system begins the game feeding the player only “golden tasks”, which we know the answer to. Once the player demonstrates that they are actually trying to play properly, the ratio of these verification tasks gradually lowers. This process is completely invisible, so even if spammers understand the mechanism, they will not be able to cheat it.

Other challenges included thorny mechanical issues like deciding the type of gameplay to implement (the first prototypes required input methods other than typing which proved to be very inefficient), issues with the number of parallel players to crosscheck answers, and scalability of the system. There were also more mundane issues, for example some people were unwilling to use their Facebook account to connect with the games (a very vocal minority requested login by email, so we introduced it a couple of weeks after the launch).

Holy moley: some staggering statistics
Figures for Digitalkoot are as impressive as they are interesting. The amount of gameplay for each person varied enormously, from a few seconds to more than 100 hours. The average was a more down-to-earth 9 minutes and 18 seconds. Men have proved more eager to join the cause and better at topping the charts (the top four players are men), while women completed on average almost twice as much tasks than men.

At the time of writing 55,000 people (and perhaps one dog) have taken part in the experiment. This in a country of only 5 million (people). Together they have contributed 3,400 hours of their time on a voluntary basis, and achieved a staggering 99% of accuracy in the transcription of the Aamulehti archive.

Tasks for the future
Although the project has run extremely well, there is a lot we can improve on. Next time around, for example, we would like to: minimize redundancy while checking the accuracy of tasks; be able to distinguish between words belonging to a title or to the text body; add soft keys for keyboards without the letters “å, ö, ä”; add new types of microtasks (typing is not the only thing we can do, of course); improve gameplay mechanics and reward mechanisms working side by side with professionals (or figure out in which contexts the game interface undermines efficiency instead of increasing it).

One thing that is apparent from the long list above, is that we here at Microtask will not be running out of tasks any time soon.