Exploring Artificial Intelligence (AI) in Software Test
Our CTO, Jim Zuber, has been busy exploring the use of artificial intelligence in software test. He attended several testing conferences and came back with a few observations about the use and challenges of using AI in software test.
Over the past few months, I have been investigating how Artificial Intelligence (AI) might be able to enhance the testing products and services we offer our customers. This investigation included drilling into 20 testing products that claimed to leverage AI, getting our hands dirty developing some simple AI models, and attending three conferences with a specific focus on AI presentations and tutorials including:
- Techwell’s Starwest Software Test Conference
- Pacific Northwest Software Quality Conference
- O’Reilly Artificial Intelligence Conference
Although our AI strategy is still in its infancy, we are convinced that Artificial Intelligence is going to have a transformative impact on not only software testing but society in general.
AI is currently being used in software test to do the following:
- Automatically identify the purpose of various dialogs and controls on web and mobile applications. This includes using AI to weigh various identifiers for GUI objects (buttons, input fields) such that, automation scripts are made more resilient.
- Automatically exploring application pathways using an AI model called reinforcement learning. This means that most of the obvious pathways that you might want to test on an application can be self-discovered by AI.
- Analyzing bug databases, compilation logs, website logs to determine what needs to get tested, whether tests are flakey, who is the most effective programmer, what code modules are most likely to have bugs, and a host of other things.
How will AI Change the Software Testing Landscape?
You don’t necessarily need to be a data scientist or AI expert to implement artificial intelligence in your products. Google and others are making amazing AI tools available to application developers. A great way to see what is available off-the-shelf is to go to Google’s Cloud Vision product and drag an application screen shot into the “Try the API” area and check out how much info is available about the image in the various results tabs.
AI testing is not like the testing most of us do today where we might put two specific items in a shopping cart, then validate that the total price adds up to the right number. AI testing is more generalized such adding two items to a shopping basket, deleting one, then successfully checking out. It’s about verifying the functional flow and less about more formal black and white conformance checks. It’s an 80% solution. However, if you think about DevOps and continuous release cycles, an 80% solution may be all that is practical.
Concerns About Testing AI Capabilities in a Product
Another area to ponder is how to test AI capabilities embedded in a product. If you are recommending products to customers based on their past purchases, how do you know your AI engine is making the right recommendations? Check out Angie Jones’s great presentation on how she approaches this kind of a challenge.
Another concern is AI engines with unintentional biases based upon the data that was used to train the AI engine. You certainly wouldn’t want a bigoted chat-bot interacting with your customers! There have been some spectacular fails in this area that embarrassed some very big companies.
Hackers will be working hard to leverage AI in diabolical ways. How about a little deliberate graffiti causing a self-driving car to see a stop sign as a speed limit sign. Or how about pasting pixels from one person’s face into the image of another person such that the AI engine misidentifies the images? I saw demos of both these scenarios. Scary and challenging to test!
Further Exploration of Artificial Intelligence in Software Test
If you are interested in exploring AI further here are a few interesting places to visit:
- TensorFlow has an app where you can turn the knobs of a deep learning neural network. You can watch it in real time as it attempts to come up with classification boundaries for a set of data.
- Go to Halite to see AI engines from 100 countries battle other AI engines to the death. It is really more of a learning exercise in the form of a video game. Nevertheless, it is an illustration of how people all over the world can harness AI in different ways to accomplish a similar objective – (destroy your opponent!)
- Kaggle has an app where users attempt to build AI models to predict who survived the sinking of the Titanic based on the information available on the passenger manifest. Things such as gender and location of birthing make a big difference in survival. It’s astonishing how accurate some of the AI models are in this competition.
I would enjoy hearing from customers who have started in their own journeys leveraging artificial intelligence in software test.
Author: Gary James
