Automated Testing or Manual Testing for SaaS based Products?
SaaS product testing – where defects impact every paying customer and changes are rapid-fire – poses unique and daunting challenges. The speed of test automation is appealing, but what specific challenges does automating testing bring?
Sanity/smoke tests (testing to make sure there is no collateral damage in apparently unconnected code) are a prime application for automation in SaaS. An automated test suite that exercises all of the major features of the system each time a new revision is released does a lot to catch the kind of bug that can bring users to ask, “Didn’t they test this thing?” Design your automation to check everything for basic functionality without going too deeply into any one feature’s parameter stack, and then flesh it out with links to regression test suites for prior feature releases and defects.
That said, manual testing will still be needed to verify that everything not only works but is user friendly enough to remain usable to humans. Too much reliance on automation removes the focus on the fact that the customer/user has to be able to intuitively understand the system interface and the uses of its controls. Use techs to test the usability and, where possible, use the computer to test predictable functionality.
Check out the white paper for a full discussion of manual versus automated testing and where each excels.