Testing Times

March 2nd, 2011 by

microtask_testing_timesLike life, love and the lottery, making mobile phone apps is a gamble. Play your cards right and you’ve got Angry Birds. Get it wrong and your app is left in the store with a two star rating, gathering virtual dust.

Back when Apple’s iPhone had an effective “app monopoly”, developers had it relatively easy. Okay, there was always the problem of getting stuff accepted into the app store, but apps only had to be designed for one interface, one handset, one puritanical CEO. iPhone apps still dominate world sales but now Android, Blackberry, Nokia and Microsoft are all focusing on smartphones. These days apps have to work on multiple phones and carriers, fighting for an edge in an increasingly crowded market.

Mob Rule?
At some point, everyone experiences that “cry-with-frustration” feeling you get from badly designed software: websites with no “Home” button, buggy videogames, apps that seem to be made for cinema-sized screens instead of mobiles. As every software engineer knows, all programs are supposed to go through “beta” or user testing before they get released. The trouble is that testing is often rushed, done “in house” (rather than by real users) or just ignored. With mobile apps, testing is particularly expensive and time consuming. Companies have to buy hundreds of handsets and then recruit hundreds of testers to get any meaningful results.

The situation clearly has crowdsourcing potential. Using the crowd, developers can test apps globally, accessing thousands of testers and hundreds of different devices. Several companies now offer cheap, crowd-based testing services.

One is the Canadian startup Mob4Hire. Otto and I met Paul Poutanen, the founder of Mob4Hire, last year at CrowdConf. Poutanen, who has Finnish roots (the “-nen” ending in his name is a dead giveaway), presides over a community of 50,000 testers and 1400 developers. Mob4Hire offers a range of testing services, from usability to localization, via a paid, mobile crowd. Testers run apps using their own phones, meaning the company can claim access to 30,000 different handset models.

Equally impressive, Poutanen also has an excellent grasp of important Finnish drinking terms including “kippis”, “Pohjanmaan kautta” and “hölökynkölökyn”. (Lest I am accused of any Finnish bias, I should mention that Mob4Hire isn’t the only company offering crowd testing. Another major player is uTest which raised an impressive $13 million in a recent investment round.)

App-ealing to the Crowd
As someone who was once in the mobile business, I think it is clear that industry standards will benefit from crowd testing. As well as being quick, cheap and easy, the crowd could also create a lot of free buzz around new apps. Mob4Hire is also pushing its Mobstar rating, as a kind of crowd tested, gold standard guarantee of app quality (is it just me, or does “Mobstar” sound like an app award sponsored by Tony Soprano?).

This is not to say that such testing services are infallible, even with the mob behind them. Like with any creative product, there will always be an element of unpredictability in how the market receives an app, even after crowd testing. Just as some books and films flop on first release then turn out to be enduring hits – The Big Lebowski waltzes to mind – so too may apps that test poorly.

On a final note, given that crowdsourcing itself plays a role in increasing numbers of mobile apps (think of Waze, the crowdsourced traffic info app), it’s surely only a matter of time before the crowd testers end up rating a crowd based app (metacrowdsourcing?). I guess the question is, how many stars will the crowd give the crowd?

  • Paul Poutanen

    Thanks Ville!nnWhat we are really finding at Mob4hire is the marketing research aspect which is probably as equal in importance to functional testing.nnWe say at Mob4hire n1. Don’t Crashn2. Don’t Suckn3. Get StickynnBrands that are moving into a mobile strategy need to get real people’s feedback both before and after launch.nnThanks again Ville!nnKittos!nnPaul PoutanennnPresident and Founder nnMob4hire Inc.n