The 4 Surprising Benefits of Regression Testing
Regression testing is often viewed as the ‘red-headed stepchild’ of software QA. This is unfortunate as regression testing can contribute more to product quality than most would expect.
The main purpose of regression testing is to verify that the problem corrected by a code fix is actually corrected by that fix.
A less common perspective is to use these tests to verify that other operational aspects of the system have not been compromised by the update that was intended solely to fix a known defect.
Tests are compiled to perform these verification’s and are usually archived in the code control system with the code updates themselves. This ensures that they are run when a system version is assembled for a new release.
These tests are then aggregated into suites and used to verify the effectiveness of the various fix updates in future releases that are made to introduce new features or fix bugs. The suites can also be incorporated into automated comprehensive tests that verify system operation for each release generated by the code management system.
Top 4 Finer Points of Regression Testing Explained
As with all aspects of software QA, regression testing has a number of fine points that can enhance the application and benefits of the process.
1. Defect Management Definitions
Regression tests typically start out as part of a bug report to the defect tracking system. They begin as instructions for replication of the symptoms caused by a code defect.
These step by step directions are intended to cause the system to exercise one or more features with specific data inputs that yield the undesired behavior in a consistent manner that facilitates troubleshooting by development engineering.
Replication instructions are the beginning of a regression test for the bug fix. They provide the process for generating the conditions of the defect symptom. The addition of a verification process to assure that the symptom has not re-appeared completes the regression.
2. Thorough Symptom Coverage
The process of massaging the replication instructions into a regression test offers the opportunity to try variations of those instructions to discover and examine additional symptoms. Many defects generate collateral issues with other parts of the system code that are difficult to associate with the bug that actually causes them. By the same token, the effort to fix the defect, causes unintended side effects related to those same undetected connections.
Working with the regression aspects of the test can reveal these connections and also point out process or input data variations that produce unexpected symptoms to appear. Adding these modifications to the test will cross-check for these symptoms in future releases and should be considered as source material for additions to a system sanity check that is specifically intended to check for unintended side effects of code modifications.
3. Test Documentation
Even more than most other software development efforts, regression tests need to be thoroughly documented. They are a primary communication channel between QA and the development group about what the problem was, what was done to fix it and what was done to verify that fix. Fully explaining the resulting regression test in the defect management report documentation, recording the details of the test in a regression log and detailed comment blocks at the start of any test code are golden.
Regression test documentation can become a useful management tool, especially when it is resident in the defect tracking system. Such a system, thoroughly integrated with the code management system, provides to the point release documentation and can be used to produce automated release documents.
4. Structure Conformance
The best way to conduct regression tests is via test automation. Automated testing has been promoted as a panacea and reviled as a waste of development resources but, as usual, the truth lies between these extremes.
Automation, by its very nature, promotes rapid continuous test iterations and these are the best way to be sure that a system release has been completely wrung out for defects before the ultimate test team, the customer base, gets their hands on it.
A software development group will have its preferred automation framework and a set of test design criteria that support it. Automated regression tests should be designed from the ground up to work within these criteria and on the automation framework. This will ensure that regression test suites are used and re-used to guarantee system operation and that they can be incorporated into automated sanity and smoke tests as well.
Regression Testing is a Key Aspect of Software QA Operations
It shines a light on the way forward for the defect repair effort and, properly executed, it supports both code development and overall system operational verification.
Contact us today for more information about how regression testing fits into our testing process.