Better shred than read: DARPA uses competitive crowdsourcing to revive destroyed documents

January 12th, 2012 by Ville Miettinen

As if things weren’t already hard enough for them, crooked bankers , deposed dictators and international super villains have one more thing to worry about. Having spent their last hours of freedom shredding incriminating evidence into neat strips, they might have thought they could get away with their misdeeds. It turns out however that those shredded documents might not be as unreadable as they thought.

Our old friends at DARPA (or the Defense Advanced Research Projects Agency if you have the time) decided that there might be something worth reading on those strips. They wanted to create software that could identify scanned strips and piece them together as quickly as possible.

Money to learn
DARPA’s annual budget is $3.2 billion. They could have hired a crack team of programmers without making a scratch on that figure. But even the best programmers in the world are only going to come up with one solution at a time. Knowing that crowdsourcing would provide a variety of possible solutions, DARPA decided to establish a crowdsourced competition.

With a first prize of $50,000 (much cheaper than hiring programmers) the “Shredder Challenge” had five challenges of increasing difficulty. In a clever bit of gamification, each document consisted of a puzzle that could only be solved when the document had been stitched back together. Entrants won the prize by being the first to submit all the correct puzzle answers.

“All your shreds are belong to U.S.” was the meme themed team that got their entries in ahead of everyone else and a full two days before the deadline. The team of only three San Francisco based programmers had just 35 days to complete the task. They spent 600 man-hours spent building the algorithms which made suggestions of shreds that might fit together.

Suspicious finds
DARPA claims they came up with the Shredder Challenge for soldiers to use in the battlefield (presumably for when they find that bunker full of shredded MapQuest directions to the W.M.Ds) and also uncover potential vulnerabilities in U.S. Government document disposal practices. Conspiracy theorists will surely claim it’s only a matter of time before the software is turned on us ordinary decent folk with nothing to hide (except maybe some irregular tax returns).

Should we throw out our shredders then? Well there is no news of a slump in shredder sales. The thing is, even if the software was cheap and freely available someone still has to get hold of the paper shreds, laboriously scan them all, then reassemble them after the software has worked its magic. You would have to have something pretty important in those shreds (pirate treasure maps?) for someone to go to all that trouble.

DARPA’s intentions seem pretty straight forward. In fact for a government agency tasked with “preventing and creating strategic surprise” they are very open about many of their projects. It’s difficult to make use of crowdsourcing without the crowd having some idea of what you are doing, which is great for the rest of us because every now and then we find out about a tantalizing DARPA project.

As we have seen before in this blog DARPA has certainly caught the crowdsourcing bug, and is well positioned to experiment with different crowdsourcing models. This is great for the industry as a whole. With DARPA blazing a crowdsourced trail, other organizations that were unsure about how to use crowdsourcing may find a DARPA method that suits their needs.