Sanity and Smoke Testing: What’s the Difference?
Sanity Checks
QualityLogic approaches sanity checks as testing the elementary functionality of all the features in a system. Creating and implementing such a test suite allows a rapid check for code breakage in areas apparently unrelated to the most recent feature additions/bug fixes.
The idea behind these tests is to touch each feature just enough to verify that it is still accessible and performs its basic function. The check carefully avoids testing any feature in depth as the intent is to have a test suite that exercises everything quickly enough to complete both the test execution and the results evaluation in less than half a day. The sanity check test suite should be completely automated and it should be triggered by the automated build system. This provides a check of each new system version as it is built.
Smoke Testing
In our lexicon, smoke tests are intended to stress specific system aspects. Unlike a sanity check, a smoke test typically drills down into the details of the feature or bug fix that it addresses. Instead of a high level assurance that the function is still accessible and does roughly what is expected of it, the smoke test verifies all of the function’s parameters with a full range of and, importantly, out of range values. Error trapping and graceful failure recovery are also of principal interest in smoke testing to see that, above all, this code aspect is robust and durable.
The name smoke test came from the manufacturing world. It typically denoted a test to destruction in which the end of the test was the point where “smoke came out.” In some ways, this is similar to stress testing but with some important distinctions. Stress tests typically use normalized parameter values and invoke the stress aspect of the test by running many instances/users for long periods of time.
Keeping an updated system sanity check in place is an invaluable tool to avoid unexpected consequences degrading what otherwise would be a successful code roll out.
QualityLogic offers a full range of software test services. Contact us today to start improving the quality of your software.