Continuous Testing: A Journey of Ongoing Improvement and Effort

Technology has fundamentally changed the way businesses operate. Remaining relevant and gaining the competitive advantage is top of mind for most technology business executives. One of the ways businesses are attempting to stay competitive is by accelerating their release cadence. Adopting the tenets of Agile or DevOps is making it possible to develop and release software at a level that meets user demands.

Calling this a drastic change from the waterfall method of taking months to create a new release and more months to test it is the very definition of an understatement. Note that this sudden dramatic acceleration of development and release cycles did not make testing go away.

The head-long release of new software versions goes out to a software marketplace that lives and dies on user reviews. The egregious defect that might not have been caught before a patch could be sent out can now drive hoards of negative user reviews within hours. Therefore, we now have Continuous Testing (CT) to go with the accelerated development processes listed above.

Test automation has taken a curious path from being a good idea, to a cure-all, to not worth what it costs, to a matter of marketplace survival. It started out as a UI-centric tool that could populate screen forms quickly and often and then actuate all the controls on that screen. But to support continuous testing for Agile/DevOps, test automation will need a substantial shift in focus.

Why Modern Development Methodologies Demand a Continuous Testing Approach

Quality as a concept is predicated on the careful, deliberate examination of the object under est. This has led to testing processes remaining stuck in the past even as organizations have invested heavily in high-speed development. Test teams have been showered with automation gone wrong horror stories and they want to avoid the pain for as long as possible.

That said, Agile and DevOps have been adopted by over 80% of the industry and that implementation absolutely requires Continuous Testing. As software delivery accelerates, testing cannot keep pace if it relies on manual processes. It is not possible for manual testing to provide Agile developers with the verification that high velocity releases require.

True Continuous Testing cannot be based on simply taking a manual quality process and adding in some automation. And, without a reliable QA safety net, Agile goes from a market advantage to business risk.

Continuous Testing Gains Leverage

Agile QA testing is typically viewed as a progression from Unit Tests through Unit Integration on to System Integration ending at User Acceptance Tests. Continuous Testing pushes the primary focus toward the bottom of that list. UI based tests, typically performed in System Integration and UAT, require environments that are simply not available early in development. As these services and data sets are refined and finalized, any automated UI tests that are dependent on them begin to fail unless they are re-written to account for these changes.

The proliferation of APIs as the system communication method of choice has provided a handy lever for test automation. CT requires automation and drives toward testing at the API layer. APIs are considered the most stable interface type in the system. They are available earlier than UI controls and offer connection to back-end and third-party functionality that UIs do not. Best of all, they directly lend themselves to machine testing.

Automated API Testing

Two vital supports for API testing are Service Virtualization and Test Data Management. Test stubs for cloud-based services and detailed, quickly reset test data sets enable continuous early testing and facilitate defect reproduction. This moves discovery and correction closer to the start of the development process.

A set of sanity and regression tests that are triggered off code check-ins and builds should be used as release gates to guard against collateral damage. Too often, the rush to create features and fix defects causes damage to the system in unexpected places that have obscure connections to the code that changed. Risk coverage optimization through carefully maintained sanity and regression tests is the best prevention effort to avoid this.

Behavior Driven Development

One way to merge Continuous Testing into Agile/DevOps is to institute Behavior Driven Development. Too often this is confused with Test-driven development which focuses on the developer’s idea of how the software should work. Behavior is how the user expects the system to behave. As Agile development stories are created, use the Given, And, When, Then parameters to clearly specify what the behavior of the test, and by definition the code, is supposed to be.

This approach combined with test automation of the lower levels of the test tree should bring the best CT results with the least pain.

The Continuous Test Journey

Many QA, Development and Product managers have viewed this task with despair. A better approach is to realize that Continuous Testing is a journey rather than a destination. Continuous Testing goes hand in hand with continuous improvement and ongoing, consistent effort pays off:

Google’s John Micco says that they have 4.2 million individual tests running continuously both before and after code submission with 150 million test executions per day and 99% of all these test executions pass.

Interested in Learning More? Let’s Talk.