Fast Food and the Crowdsourced Software Testing Phenomenon

Crowdsourced usability testing has marketing appeal. They hook you with mentions of the big players…Adobe, Jet, Oracle, and more. On the surface it looks good. You simply sign-up (and pay a fee), submit some questions, and your tests are magically sent to on-demand users who offer feedback. What could go wrong?

Crowdsourcing your software testing is like eating fast food.

Sure, it can taste good, but is it the best thing to put on your plate day after day?

Crowdsourced software testing promises right now, and real fast, user feedback. But will it truly serve your best interests? You developed your software based on two things: a brilliant idea — and the desire to generate revenue. To ensure adoption — and revenue — you need test feedback that’s more like a four-course meal at a five-star restaurant! Can you trust random internet users to offer educated feedback, and can you trust yourself to create usability tests based on a deep-dive into testing methodologies?

Let’s talk about this…

What is Usability Testing?

Let’s back up and define usability testing. The cut and dry take is this:

Usability testing is a method that measures an application’s ease-of-use from the end-user perspective. It is considered ‘non-functional’, because here you’re testing how the app operates, rather than ‘functional’ aspects like how the app integrates with hardware. Usability focuses on the application’s ease-of-use from the end-user perspective and is often performed during the system or acceptance testing stages. The goal of usability testing is to determine whether the user interface and aesthetics of an app meet the intended workflow for various processes, such as logging in. Usability testing is a great way to check that separate functions and the system, is intuitive to use.

System testing is the holistic overview. It evaluates the complete system to ensure it meets specified business and user requirements. Yes, this is best done near the end of the development process. Acceptance testing is the last phase of functional testing and is used to assess the software as ‘ready for delivery’ You might know this as Beta testing.

Both system and acceptance testing typically take place near the end of the development cycle. This timing seems to make crowdsourced usability testing a good fit, right? Not necessarily! There are many benefits to be had by wrapping user testing into earlier dev processes. Today’s software development life cycles (SDLC) are dominated by Agile and Continuous Development methods based on quick iterations and releases. As the pace of software development intensifies, there are many advantages to including usability testing in these ‘shift-left’ ideologies

Issues with Crowdsourced Software Testing

Crowdsourcing testing is an after-the-fact approach to app development. The most heinous examples of crowdsourcing can be seen in the app market. We’ve all experienced downloading an app that sounded great, but in reality, was so buggy we uninstalled it after one or two attempts at use, right? Some developers think they can skip formal testing altogether by having actual users do testing with their live app! By skipping formal testing, they force their revenue-generating users to become their unwitting testers. This is not a happy place for a user to be! In this scenario sometimes the app survives, but most of the time it doesn’t.

Ok, so none of us wants to be the homegrown developer who releases untested apps to market. What then? The next level of testing comes from websites who claim to manage your testing for you. One popular website for crowdsourced testing proclaims “In just two hours, capture the critical human insights you need to confidently deliver what customers want and expect!”

This sort of marketing spiel makes me uncomfortable. I start asking myself: “Seriously? Results in 2 hours? Do these companies have people just sitting around waiting for me to decide to test my software? Who are these testers, anyway? What’s their experience with testing, and can they articulate feedback that means anything? Or, are they just clicking around trying to break my app or website?”

Then, a little bit of panic sets in…do I know how to design a user test? What sorts of questions should I ask, or user pain-points should I look for? If you look at the steps necessary to become a “tester” for these crowdsourced platforms the marketing often reads something like “No special knowledge necessary!”. But stop and ask yourself, “is this who I want testing my software?”

I don’t know about you, but if I went through the trouble of building software to enhance my business revenue, I want testing that’s thorough and professional — not rushed like a double-double with fries at the drive-through!

Benefits of Professional User Testing

Considering the potential pitfalls of crowdsourced user testing helps us understand the benefits of hiring a quality assurance company. Real QA experts know what the most common causes of app or website failures are. They are technology superusers and have strong backgrounds in testing methodology. They are NOT random internet users! And these are the folks I want testing my software.

At this point, you may be wondering why enterprise-scale companies like Adobe or Oracle would consent to have their logos used on crowdsourced testing websites. I know I was!

After following the proverbial rabbit down quite a few holes, reading whitepaper after whitepaper, here’s my conclusion. Large companies crowdsource their software testing for one purpose only. That purpose is to get surface level feedback after updates or new features are added. There’s also a caveat, they do this by having their very large, and very experienced, QA team create the tests they submit to the online crowdsourcing platform. And that’s the kicker. The usefulness of crowdsourced software testing is only as good as the tests that are designed. To design ‘good’ tests, you need to have expertise in testing.

So yes. big names do occasionally use crowdsourcing to verify a small iteration. But, they don’t use crowdsourcing as their primary means of testing end-to end.

A final benefit of using an actual software quality assurance company who is under contract with you to provide testing is the ability to ‘wrap QA into the build’. Today’s crowdsourcing platforms often tout ‘testing at the speed of development’ as a benefit. But is crowdsourcing truly different, or better than, managing your QA with a team dedicated to doing exactly that? Real quality assurance companies commit a team to working with your developers through the SDLC. The Agile and Continuous Delivery methodologies doesn’t change the fact that QA experts are…well…experts. And that expertise includes in-depth, practical, and specialized knowledge of ALL forms of testing, including User Experience and Usability testing.

So, what are you in the mood for now? The drive-through? Or, the 5-star restaurant?

Contact Us