3 Ways to Mitigate the Risks of Automated Testing

Test automation holds the promise of enormous benefits for improving product and service quality. It applies computational power to what it does best, the repetitive exercise of software systems. And, like any other powerful tool, misusing it is fraught with risk.

Software development has progressed from the Waterfall method that allowed months and even years for creating, assembling and testing a software release to Agile and Continuous Integration that crank out releases in a week or two. Along the way, software quality assurance had to keep up or get run over. This has created a glut of software products that have been lightly touched by testing only to be slammed by customer reviews.

The rapid release of features and defect fixes in software products and services means that test verification has had to change dramatically to keep up.

Software Testing Processes of the Past

In the past, software testing was the province of a department staffed by manual testers whose job it was to take marketing’s feature lists and verify that they were faithfully replicated in the products. Some of these people were code engineers who understood the system well enough to perform white box testing on it and ferret out deeply buried bugs. Most were techs who could follow test plans and use the system user interface to exercise all that it was supposed to do.

Any test response that looked inappropriate was logged in a defect management system and broadcast to the rest of the organization in a periodic bug report. These reports would be discussed and turned into prioritized action items at meetings of engineering, marketing and quality management. Testing and verification moved in lock-step with a glacial development methodology.

Automated Testing Necessary in an Agile World

Now, quality has been moved onto the front lines. Testers and code developers are expected to work in tandem fixing bugs as quickly as they are found. The Agile SCRUM has stand-up meetings daily in which the ‘stories’ selected for the current two-week sprint are discussed in terms of the day’s progress in both code development and operational verification. Once a fix or feature is integrated into the system, it is expected to be verified and released in a day or two at the most.

This has made automated testing a necessity. It has also made it appear to be the cure-all for getting verification done at or beyond the speed of development. While it offers vast benefits for testing system functionality quickly and repetitively, it is not the cure-all for test development. Test automation needs to be a tool in quality’s kit rather than the default test mode.

Test Automation Gone Awry

As with product code development, the test process to be automated must first be fully understood. Attempting to automate an ill-defined array of tests results in chaotic test results that rapidly become meaningless. This commonly happens when an attempt is made to automate manual exploratory tests which are guided more by the experience of the tester than a fixed test case.

Test automation is a matter of taking tests that are well defined with clearly defined result requirements and making them machine-executable. Automation will not fix problems with misunderstood code functions and product features for which detailed, workable testing has yet to be developed.

Automated Test Script Maintenance is Key!

Automated test scripts can be easily generated by test frameworks that will record UI actions and replay them. Unfortunately, such scripts are brittle in the extreme, and, if anything changes in the code, the script will yield nothing but false errors. Over-use of poorly designed or maintained test scripts eventually leads to the automation process being generally rejected as being more trouble than it’s worth. Automation requires the commitment of scarce code development talent to script maintenance and these people will be constantly pulled off to write revenue-generating code instead of working on tests.

Both of the above are going to be aggravated by the concept of ‘if a little automation is good, a lot is better.’ Attempting to automate testing of system aspects that change either on a regular or erratic basis will cause the scripts to become either obsolete or perceived as code talent sinkholes. While automation can do wonders with scripting some test processes, it is a disaster with others. It is not a one-size-fits-all.

Navigating a Safer Path to Test Automation

What to do? Test automation is both valuable and necessary and will provide the boost that the QA process needs if a few simple guidelines are implemented.

1. Management Buy-in

Get management’s buy-in to test automation as a planned process effort with an adequate budget. Test automation is going to require script maintenance and it will be expensive. The payoff from a well-designed and maintained array of automation scripts is worth its price but worth nothing if it is abandoned because the budget dried up.

2. Choose the Right Automated Test Tools and Frameworks

Determine before starting that the tools/frameworks to be used are suitable to what is to be automated and how that part of the system under test works. The framework on which test scripts are built must provide the flexibility and depth of functionality to stimulate the system under test and correctly record the results that it is expected to see. And expect to spend money for staff training on the automation tools as it will pay for itself many times over.

3. Carefully Determine What Tests to Automate

Layout which tests are to be automated because they are repetitive and lend themselves to this type of testing. Avoid the idea that all manual testing can be replaced with automation. This is how automation projects turn into costly failures. Automating user interface tests means that the scripts must change each time the interface is changed which will make the scripts as expensive as the UI code. The opposite end of this is that automating tests of API calls is an excellent use that puts the computer testing another computer.

A simple spectrum of automation viability is:

Test automation is a powerful tool, but it won’t do everything. Plan it, manage it and fund it properly and it will be a major help in navigating the Agile development crush.