6 Major Pitfalls of Crowdsourced Software Testing
If you’re considering crowdsourced software testing, you may just want to hit the pause button and read this first.
The crowdsourcing concept has been growing in popularity for quite some time. These days, you can crowdsource everything from a new product to your kids’ batman-themed birthday party.
While crowdsourcing can have a role to play in QA software testing, you may want to be aware of the pitfalls. Some are obvious and some subtle, but they can all cost you time, money, and reputation.
Crowdsourcing Has Its Limits
Wikipedia (itself a crowdsourced knowledge base) defines crowdsourcing as “a specific sourcing model in which individuals or organizations use contributions from Internet users to obtain needed services or ideas.” The crowdsourcing business model has been a successful tool with a wide range of uses, from gathering and compiling information (Wikipedia) to funding personal projects (GoFundMe).
Wikipedia is both crowdsourcing’s biggest success story and its worst cautionary tale. Input from hundreds of thousands of “Wikipedians” has created an incredibly large, diverse body of knowledge. But not all contributions are trustworthy. Pranksters and trolls abound, making actor Jeremy Renner a velociraptor, Ernest Hemmingway the author of the angst-ridden Spot the Dog, and “reality” itself “becomes a commodity,” thanks to Stephen Colbert fans.
While it works well for many applications, crowdsourcing does have its limits.
What is Crowdsourced Software Testing?
Crowdsourced software testing is the process of sending a software product out to many individuals who are networked by a company that offers their QA software testing as a service. Literally, the product is sent to people who attempt to operate it through its user interface and then report back to the service company on what they find. The service company is the QA tester’s interface with the product developer and presents this client/developer with a defect list compiled from the testers’ reports.
This method of software test and verification has become popular due to its perceived cost advantages over traditional test processes and the opportunity for the developer to hand off test process management to a third party. It does, however, have a few drawbacks.
The 6 Major Issues with Crowdsourced Software Testing
While crowdsourced software testing provides the opportunity to have a large, user-based cross-section of QA testers exercise a product under their own conditions, it also has some distinct drawbacks.
1. Only Available at Product Release
Since testing must be done through the product’s user interface, crowdsourced software testing can only be done at the end of the development cycle. The code must be sufficiently integrated for the user interface to operate all the features to be tested, and that requires a product that is very nearly complete.
This means that crowdsourced testing is of little value at the beginning of the product development process when defects are least expensive to correct. The advantage of a third-party software QA service is blunted by the necessity of an in-house or additional service to perform early development stage testing.
The same issue also means crowd-sourced testing can’t participate in an Agile development process. There is no way to integrate the test personnel directly with small teams of feature-focused developers to get QA operating as part of the development process. (See “Agile Testing Services Tip: Break Down the Barriers between Development and QA” to find out how this integration works and the benefits it provides.)
2. Exposes Early Product Version for Poor Reviews…
In this age of instant Internet communication, products rise and fall very quickly with online product reviews. Mobile device apps, in particular, are vulnerable to user ratings that tend to be based on initial impressions rather than in-depth considerations of product operation.
Non-Disclosure and other information confidentiality agreements aside, putting a new product that likely still has some significant bugs out to crowd-testing can get unwanted user reviews of that version. The client’s actual control over who sees the product and what they do about it is minimal at best. Products in extremely competitive market segments are particularly vulnerable to technology reviewers’ efforts to search out early release versions and scoop their own competitors.
3. …And Y
our IP to Pirates
As with #2, intellectual property is vulnerable to NDA enforceability and enforcement efforts. One of the biggest benefits of crowdsourcing, having the product tested all over the world, is also its biggest risk. Use of offshore QA testers keeps crowdsourced software testing costs down but where IP laws are not enforced or non-existent, there’s no guarantee that pirates won’t cannibalize your market.
4. Large QA Tester Pools Produce Redundant Defects
Set a large number of people to examine the same object and they are likely to see exactly the same aspects of it.
A good defect report provides value to development by reporting a high level of detail. The service organization has a primary task of filtering redundancy out of the defect reports. But that filtering effort reduces the usefulness of detailed defect reporting by reducing the redundant reports to their least common denominator. This puts the client in the position of either accepting diluted defect reporting or parsing the actual defects out of a high information noise environment. A flood of reports on the same defect can obscure the one report of an unusual observation that is genuinely valuable.
5. Direct Analytical Regression Tests are Difficult
One of the most common aspects of software system verification tests is the test/fix/test cycle provided by regression testing. QA engineers and/or technicians test the system, document defects and, if necessary, confer with the development group on exactly how to replicate them. Development works up code patches (or entirely new code segments) to correct the defects, integrates them back into the product and then turns it back over to the QA personnel to re-test and verify these fixes.
The operational value of regression testing comes from being able to minutely examine the causes and symptoms of each defect – known as exploratory testing. This enables the development group to release their ‘fixed’ system to a QA who is expected to verify that it a) corrects the flaw and its symptoms, and b) that they haven’t unintentionally damaged some other part of the system’s operation.
Crowdsourced testing puts the actual test personnel several communication links away from development and makes detailed cooperation difficult or impossible. Turning the system over to a new group of remote testers makes focused regression tests that, in some cases involve white box testing, a considerable challenge. Test results that contain new defects then become a question of “did the first test group simply miss these,” or “were they caused by the new code changes?”
6. QA Tester Pool Variability Between Release Versions
This brings up the matter of test facility consistency. As the product is revised to add features and bug fixes, it will be tested and re-tested to verify both the changes and the continued integrity of the system itself. The closer the product gets to final release the more critical the test/verification process becomes. Each defect caught is a poor review avoided.
There is, however, the problem of sorting out defects resulting from changes in the testing environment from those caused by the code. A stable roster of test personnel provides a stable test platform where observed changes come from the subject of the test rather than the understanding/execution/evaluation of the tests by the test staff. While these distinctions can be sorted out after the fact, this adds time and cost to the product development process and delays releases.
Striking a Balance
Crowdsourced testing exposes the subject code to many different operating environments, operational expectations and functional perspectives. As such, it is much better suited to post-Beta testing of product/service usability than the back and forth of functional verification and regression testing.
A software QA testing resource that is a dynamic, integrated aspect of product development is vital to maintaining the flow and velocity of the Agile process. And that development momentum is what gets the product through the teething phase of code and feature integration and into the hands of Beta testers and ultimately its user community.
Get your software off to the right start and follow through to the finish. Speak to a QA professional at +209 423 7936, or contact us online.