The Challenges of Software Testing in a Continuous Delivery Environment
By now you’ve probably heard or have even implemented Continuous Delivery (CD). What you might not have considered is how continuous delivery affects testing.
Continuous delivery is simply the process of getting new software builds into user’s hands, as quickly as possible.
Every company strives to offer a quality project in the shortest time possible, because time = money, right? But CD is a unique build environment, with unique software testing considerations. Let’s take a look at some ways continuous delivery can benefit your company, and three strategies for tackling software testing for continuous delivery.
Continuous Delivery 101
Continuous delivery aims to build, test, and release software faster and more frequently. This approach can reduce the cost, time, and risk of delivering software updates. Continuous delivery reduces the risks associated with software releases because each code change is already tested and releasable, if testing is integrated properly.
Releasing software updates is notoriously time-consuming and, sometimes, quite a painful process. Continuous delivery promises to turn an organization’s update-frown upside down! The primary benefits of CD are:
• faster user feedback,
• quicker time to market,
• increased quality, and
• improved customer experience.
Developers tend to appreciate continuous delivery because of the rapid feedback on their code. When a new feature is released they know quickly if they’ve introduced an issue. What developers don’t appreciate is testing approaches that don’t keep up with continuous delivery!
Faster feedback leads to increased productivity. Rapid and reliable releases lead to happy customers. Both lead to increased revenue. Continuous delivery is a win-win for organizations and customers alike, so what can go wrong?
Software Testing for Continuous Delivery
Developing a testing strategy for continuous delivery can be a struggle for some testers. In CD, the focus is on:
• Test automation
• Delivering features in small chunks, often without a user interface to test in
• Features developed in a matter of hours or days, not weeks or months, with little time for planning and preparation
• Time to market over perfect software
These considerations ask testers to learn some new tricks. Since the software is delivered continuously, testing needs to be continuous also. This means software testing for continuous delivery happens throughout the build/deployment pipeline. Testing gets integrated into every aspect, from business requirements to production.
New tricks, critical thinking, and communication are a testers best friend in continuous delivery.
Critical Thinking and Communication
It’s all too easy for a tester to make the mistake of focusing exclusively on technical considerations. Especially when high priority is placed on automation. It doesn’t hurt to step back and think about the larger picture. Is the team building the right thing in the first place? Are processes working or could they be improved? How should you approach non-functional testing, security, performance and accessibility?
These considerations and more rely largely on the soft-skill of communication. A tester who is confident discussing issues ranging from bugs to user-stories is a valuable asset to a continuous delivery team.
Layered Test Strategies
Talking about bugs and test plans is one thing. It’s another to talk about how you’ll build testing into the development process. We suggest employing a layered test strategy model for continuous delivery. Consider something like this:
• Exploratory testing
• Automated regression testing
• Automated integration testing
• Unit testing
Here, testers need to think about how they’ll support developers through this process, and especially what it will take to automate large portions of the testing. Identifying what kind of test coverage and infrastructure you’ll need will allow you to deliver the quality your customer expects. Thinking about and designing the test environment offers support for developers, and value to the product.
Exploratory Testing and Testing Tools
Shifting from automating tests, we also need to look at business needs through exploratory testing. Exploratory testing is different from scripted testing. In short, it’s an “intellectual process” that combines test design and test execution. Exploratory testing begins with the creation of a mission statement for the software. This could take the form of set of heuristics or ideas, a customer journey narrative, or persona testing based on user needs.
However exploratory testing is articulated, there are several key things to keep in mind:
- This is continuous delivery, so keep the tests lean
- Provide enough information in the mission statement for someone else to jump in, if necessary
- Document testing outcomes.
There are many tools to make exploratory testing faster, easier, and more effective. Tools to help you interrogate and exercise your system. Automated scanners to identify flaws and vulnerabilities in your solution. Tools to help you investigate usability, accessibility, portability, deployability and any other ility’s you can think of! CD testers must become experts in finding and using a variety of testing tools. Think of it like getting new toys to play with!
Can Software QA Overcome the Challenges of Continuous Delivery?
The continuous delivery method of development benefits companies and through frequent release cycles and reduced feedback time. A large part of the CD strategy involves adopting automated testing methods. It’s important to keep in mind that not all software quality attributes can be tested through automation alone. Therefore, not everything can or should be automated. Software testing in continuous delivery requires an agile QA team with a firm grasp of both manual and automated testing best practices.