Generative AI—Is It the Catalyst for Evolution in Test Automation?

When I think about artificial intelligence (AI), I'm reminded of a time not too long ago when AI was just a futuristic concept, something we saw in sci-fi movies and read about in books. Little did I know that AI would soon become an integral part of my daily life, reshaping how I work and make decisions.

As a software test automation lead, I've seen the testing landscape evolve dramatically. But the introduction of AI genuinely revolutionized how I approached my work.

Before Generative AI

Automation testing has been an invaluable tool throughout the software development lifecycle, enabling rapid execution of test cases and reducing the time and effort required for regression testing. Automation frameworks and command sets have played a central role in ensuring software functionality, detecting defects and maintaining product quality. New AI tools, however, with neural networks and deep learning models continuously emerge, and most of them are very effective tools that definitely improve testing.

But these conventional testing methodologies possess their own inherent restrictions. Let us delve into a few of these limitations:

• Maintenance costs: Automation scripts require constant maintenance as the software evolves, which increases the cost and effort of script maintenance.
• Limited scope of testing: Automation testing focuses on predefined tests that often struggle to adapt to dynamic changes in the software environment.
• Complex UI testing: Extensive automation of user interface (UI) testing can be complex, leading to gaps in test coverage.

Generative AI can help overcome some of these limitations, but it may also bring about new challenges.

Generative AI

Generative AI is a type of deep learning that can generate new data or content similar to the original data or content. For example, generative AI can create realistic images of faces, animals or landscapes that do not exist in reality, as well as create text that imitates the style and tone of a particular genre. Generative AI uses various techniques such as generative adversarial networks (GAN), variational autoencoders (VAE), and transformers to learn patterns and features of data and then generate new samples.

In recent years, the field of generative AI software testing has undergone significant change. Traditionally, automation testing has been the cornerstone of software quality assurance, ensuring the efficiency and accuracy of software functionality verification. However, Generative AI is poised to transform the automation testing landscape, introducing new approaches and capabilities that promise to change the way software is assured:

• Test case generation: Generative AI can dynamically generate test cases by learning how the application works, identifying edge cases and creating test scenarios that humans might miss.
• Self-correcting: Test automation tools that use AI algorithms can adapt to software changes and update test locators if they change, which reduces the maintenance of testers.
• Improved UI testing: After learning UI, Generative AI can simulate human interaction with UI.
• Larger data sets: Generative AI analyzes large data sets and generates test data with different combinations, discovering subtle flaws.
• Shift-left testing: Generative AI facilitates error detection earlier in the development process, reducing the cost and effort of debugging later.

There are a few challenges and factors, however, that must be taken into account prior to the employing of any generative AI models:

• Data Quality: Models need high-quality training data to perform effectively—garbage in and garbage out applies here.
• Expensive: Machine learning & deep learning are an expensive technology to implement. It is estimated that the resources for developing a large AI model have roughly doubled since its inception.
• Interpretability: Understanding AI-generated test cases and results can be challenging. Ensuring transparency and interpretability is crucial for trust and accountability.

After all this discussion on the drawbacks of traditional automation and the benefits and challenges of generative AI, the testing methodology and disciplines aren’t going anywhere. In fact, it creates new opportunities to build and learn new skills, tools, and roles. The requirements to optimize and go faster will never go away—and ChatGPT and generative AI help us automation testers do just that.

Below is an example which clearly states how ChatGPT or generative AI can help automation engineers to some extent, but to finish the work, the engineer needs to use their set of skills to fine-tune the solution provided by the generative AI tool.

Let’s say you want to test the UI for a web application and its browser-based app, and you decide to use chatGPT to help generate the automation script, sending the following message to ChatGPT:

“Generate selenium java script to visit ”https://www.amazon.com/” and choose the “Medical Care” option and click “See all health on Amazon.”

Here is ChatGPT’s response: 

 Image 1 

The script is close but there’s one major problem: the script is trying to click on an element which is actually not clickable, so when we run this script as it is, it will fail. An automation engineer can easily identify this issue. There are many more such examples which need human intervention and expertise. So, in this case, generative AI is capable of doing perhaps 80% of the work necessary to solve a problem, but the rest of the fine-tuned work needs domain-specific knowledge and expertise.

Conclusion

Generative AI is evolving at a rapid pace, but at this time it's not capable of replacing automation engineers. As generative AI evolves more, automation engineers need to also change their testing practices by using this technology to deliver bug-free products in a more efficient way. Will ChatGPT or generative AI eventually take automation engineers’ jobs? That we don’t know, but one thing we do know is AI will change the testing landscape forever.