What Can Risk-based Software Testing Do For Your Business?

Risk-based testing (RBT) is exactly what it sounds like, testing your software based on risk. We know from last week’s blog that circular definitions like this annoy me, so let’s break RBT down to some measurable components.

The first and most obvious question is: what is a ‘risk’? Once we clarify what a risk is in the software environment, we can discuss how to measure it and what value risk-based software testing offers your company’s QA efforts.

Jenny Bramble, risk-based testing guru and accomplished conference speaker, defines software risk as a combination of the impact of something happening (a feature failure), and the probability of that something happening. Here’s an example: You’ve developed an app that requires a login to access. If the login flow has the hiccups, the risk is your user won’t login, therefore impacting user experience and business needs (like revenue generation). Yet the other half of the RBT equation asks: what’s the likelihood of this risk, the login issue, happening? Well, both impact and probability can feel subjective, so let’s back up a step…

From my brief example you can see that defining risk based on impact and probability requires a combination of intuition, experience, and data. Defining risk, and risk-based testing in software quality assurance, may be easier with some background on the concept. So, where did RBT originate?

The Evolution of Risk-Based Testing

RBT has its origins in Risk Management, one of 10 areas in which project managers should hold competency in according to the 2018 International Organization for Standardization’s (ISO) guidelines. Risk management is useful across a broad range of industries, although methods, definitions and goals vary widely according to whether the risk management method is in the context of project management, security, engineering, industrial processes, financial portfolios, actuarial assessments, or public health and safety. Despite industry differences, all risk management strives to identify, evaluate and prioritize risk to allow a coordinated and cost-effective use of resources in controlling or eliminating the risk.

Risk-based testing is then a project management mindset rooted in the drive to understand what risks are going to have the greatest impact on a project and what risks are most likely to happen, then creating a test plan around both. RBT is used in many industries, but it’s especially useful in software because, let’s face it, there’s simply no way to test every function or code aspect for every eventuality. Any experienced PM worth their weight knows a project will have limited resources with a finite amount of time, bodies, attention and expertise. In software QA, focusing those resources where they will optimize ROI is the focus of risk-based testing. As Bramble says: “Even if we had unlimited resources, testing everything is ridiculous, impossible, and a terrible use of resources.”

Benefits of Risk-Based Software Testing

In the software quality assurance realm, risk-based testing presents multiple benefits. RBT first asks a team to define terms. This creates the framework to talk about risk and communicate cross-functionally. When all stakeholders understand terms the same way, dialogue surrounding decisions and needed resources becomes much more clear and concise (miscommunication is also a ‘risk’, right?)

Once everyone’s on the same page with terms, be they management, developers, or external users, it becomes easier to identify the risks each group perceives. This also creates engagement from all the stakeholders in a project. (and we know not having ‘buy-in’ at all levels is risky, too…)

Aligning stakeholders and creating shared terminology is just the first step. Remember that combination of intuition, experience, and data I touched briefly on? These aspects are at play in what I’ll call ‘RBT Phase 2’. In phase 2, defining and prioritizing risks is the goal. Here, the work of making a — very — educated guess about the probability of failure begins. A good place to start the process of identifying risk is by answering the following questions:

What features do users interact with the most?
How do the users interact with those features?
How likely is it the features will be used the ‘wrong’ way?
What part of the code is the most fragile?
Where have you seen failures in the past?
Are there outside influences that impact risk?
Overall, what is most likely to break?

You can see that answering these questions is one-part intuition, one-part experience, and one-part data. That’s a lot of parts for one person to offer input on! This is where phase 1 of RBT, defining terms and engaging stakeholders, pays off. It’s likely different stakeholder groups will have different answers to the above questions, generated through different experiences with either the process or product. Using these different perceptions during risk analysis will ensure that the dreaded ‘intangible risk’ — the risk never identified — doesn’t bomb your project. And, you’ll have project-peace-of-mind that those risks with the highest probability and the greatest impact have been eliminated.

Risk Analysis Tools

There are as many methodologies for defining risk, and creating an RBT strategy, as there are industries practicing risk management. A favorite in the SQA industry is the risk matrix. Creating a risk matrix is straightforward. First, determine a rating system. You could use colors or numbers, happy and sad faces, rainbows and lightning bolts. It doesn’t matter as long as you are consistent. Then, using that combination of gut-feeling, experience, and data, start at the feature level and assign each perceived case a rating.

From there you can complexify the process as much as needed to refine each risk to individual user stories, then develop test cases addressing each risk. The risk matrix can also be used to identify other risks like those arising from code fragility, integrations, and the external environment. If the matrix conveys the right information, it can be as complex or simple as you like.

Risk-based Software Testing is All About Prioritizing

Software, apps, and website all have one thing in common, they’re notorious for breaking when that one user does that one thing. Code is a complex (and often elegant) environmental interplay of man and machine, developer and user, hardware and software wherein testing for every eventuality is impossible. But, you can focus your team’s efforts to high-risk, high-impact areas through risk-based testing.

RBT offers a framework to talk about what needs tested, and what needs prioritized. Then, your team can answer other important questions like: Where do we test to get the most bang for our buck? What bits of code needs the most attention and what needs the least? What tests are a good candidate for automation? And what should we test during each stage of the SDLC?

In short, risk-based software testing provides your team with clarity and focus to derive a thoughtful testing program, leading to a better software product.

For more information on risk-based testing, follow this link.

Contact Us