Building a Resilient Software Testing Framework

Let’s be real here. I’m not a writer or natural blogger. I’ve got a marketing director who comes to me and says, “I’d like you to write about building robust quality assurance frameworks that can deal with fast-paced development.”

At this point, my brain makes computer beep-boop noises and rejects the prompt due to insufficient data to produce a decent topic. At this point, a basic GPT, of course, would regurgitate a bland response covering some high points, which might approach accuracy. What it wouldn’t do is decline to create the article because of insufficient data.

I asked some follow-up questions and got some direction on where this should go. So now, I’m going to talk a fair bit about test automation, but will also focus on the key dependencies necessary for the automation framework to succeed.

Preparing Your QA Team and Manual Testing Processes

If we want a robust framework that can support fast-paced development, the SDLC itself must be healthy. There are many questions that go into whether there is a healthy SDLC in place, and here are some of my favorites:

• How often does the QA phase delay the release?
• How often do hot fixes happen?
• Are the number of defects found in production increasing, decreasing, or staying level?
• Is the number of defects in the backlog increasing, decreasing, or staying level?
• How many test cases have been created, and how many are already automated?
• On a typical test automation run, what is the flake / issue rate?
• How many tests in the automation framework are marked as “skip” because they are flaky?
• Is the automation run tied to PR / MR in the code repository?

These questions help paint a picture of the health of the QA process as a whole and provide insights into how effective any existing automation is. When we want to accelerate and improve the test automation to increase release velocity, we have to decide first whether the ROI will be there.

Automation isn’t a low-cost investment at this time, and the best automation is built on rock rather than sand. In this case, “sand” is defined as a stressed and under-documented test process that isn’t capturing defects before production release. Adding automation to that scenario is only going to end up costing time and resources with very little to show for it.

Step 1 is getting the QA house in order. This means ensuring your QA team is staffed to handle the manual effort, establishing a strong strategy for adding test cases, and building a library of existing test cases to serve as the source of truth for the automation team. Care must also be taken to ensure that the test cases themselves are well thought out, with clear assertions and expectations, so the automation engineer can focus on developing the code rather than trying to figure out what the product should do.

Setting Realistic Timelines and Automation Milestones

Step 2 is setting reasonable timelines and milestones. If you are starting at zero, don’t expect automation to rock your QA world in just a couple of weeks. The creation of test automation code is a development effort, and development takes time to establish the core framework.

It is possible to build test automation quickly if good architecture is set aside, but then there should be an expectation that maintaining that code is going to be a bit of a nightmare. Take the time and do it right. Use good abstraction in the automation code to make those tests readable, and easier to maintain. This is typically what people call Page Object Model, but it doesn’t have to follow that precisely.

The key is logical and sensible abstraction that enables engineers to code clear and fast cases.

Establishing the Right Infrastructure for Test Automation

Step 3 is to ensure that there is reasonable infrastructure for running and maintaining the automation code.

A lot of people who have “been there and done that” like to hook that core framework with GitHub Actions or some other CI/CD framework, and shoot for an initial suite of smoke tests that can be run on every PR. That approach provides the fastest ROI for any group.

Implementing, Iterating, and Maintaining Your Framework

Step 4 is just implement, iterate, and maintain. The test automation aspect of any SDLC is only going to bring value if the test coverage grows and that the code is maintained.

Achieving Real-Time Automation Coverage

Step 5, and this really is the final step, is to bring the automation coverage to real time, so that as developers are building code, the automation engineers are building the tests.

This real time generation of coverage is where releases get truly fast, and typically reliable. When your QA engineers build a relationship with the developers so they mutually agree on selectors in advance of the code being written, there will be a magical moment when the automation engineer can actually code the test without ever seeing the development code being built. When the developer merges the code and the test just works, release is imminent. If that is paired with the manual test and a demo to the product owner, your release strategy is about as fast as it is possible to go.

At some point in all these steps, it does become possible to reduce or reassign manual QA. We have often helped manual QA teams make the leap to automation engineers. A lot of the time, though, the reality is that we simply enable existing QA team members to actually have the time needed to thoroughly test the product rather than rushing and cutting corners.

Tailoring Your Test Automation Strategy

This all may sound like a lot, and it can be overwhelming. If you want help with this process, you can reach out to us. We love working with teams to tailor a strategy that is right for their company.

These are high level steps and are general truths, but every situation has some unique elements. We’d love to help you on your own journey in building an effective and efficient automation strategy that fits your goals.