AI and E2E/Regression Testing: How to Maintain Business Control?

Designing good tests requires a strong understanding of the business. This is why QA teams, among all product stakeholders, are often the ones with the most comprehensive grasp of the business and functional aspects. This knowledge comes from years of experience and is passed on through informal discussions just as much as through the company’s documentation.

The fundamental need for deep business knowledge poses a serious challenge to the currently proliferating AI-powered solutions that generate tests from User Stories (US). Their level of performance depends heavily on the quality of the US themselves.

Yet, the underlying principle could not be simpler. We provide the generative AI with the User Stories and acceptance criteria, and it generates the corresponding test cases “at the click of a button,” properly formatted and, ideally, already automated. This works quite well, provided that the User Story can be considered in isolation. The generative AI takes the acceptance criteria one by one and transforms them into one or more test cases. To achieve good results, User Stories must be written in a clear and comprehensive manner. All the implicit assumptions and “obvious” details we take for granted must be explicitly documented in writing.

As soon as dependencies come into play, however, things become more complicated. Although chatbots are now capable of processing vast amounts of text, an LLM may lose sight of the bigger picture by getting caught up in the details, or overlook broader contexts. As a result, it is difficult to provide a single LLM with all User Stories at once.

Ideally, each US should contain all the business knowledge necessary to generate relevant tests for that specific story. Otherwise, the resulting test cases may be incomplete or irrelevant. In the worst-case scenario, the generative AI will start hallucinating and inventing the information it lacks.

Fundamentally, this problem is not new. Unclear requirements have always been one of the primary causes of project failure. To address this issue, greater involvement from Business Analysts or Product Owners would be required. This would effectively shift part of the workload from testers to these roles, which are generally more costly than QA professionals. Such a shift can be harder for them to accept, especially since it does not directly contribute to their primary responsibilities. Furthermore, this workload transfer does not eliminate the need for QA teams to review and validate the generated tests.

Epic tests and end-to-end (E2E) tests are fundamentally different from US tests. Here, the goal is to validate complex business processes involving interactions between various system functions or components. Designing such tests requires a holistic view of the system and its business context. This is when largely undocumented business know-how inevitably comes into play. Providing additional documents to an LLM is of little help in this situation, because implicit knowledge is, by definition, not explicitly documented. Nor would it be reasonable to invest significantly more time in perfecting documentation. First, because time is precisely what every team lacks; and second, because experience shows that documentation often becomes outdated almost as soon as it is written.

In short, entrusting Epic or E2E tests to an AI is like assigning the task to a tester with no experience or business knowledge. In fact, it may be even worse: while a human tester wouldn’t hesitate to admit their lack of understanding and ask questions, an AI will provide an answer regardless, even if it means spouting complete nonsense.

Beyond their granularity, number, and degree of automation, there is a fourth difference between US testing and Epic testing: US tests are significantly more volatile. When a US changes, it is typically replaced with a new one. An Epic, however, rarely changes completely. Epic tests therefore form the core of the non-regression test suite (NRT). It is crucial to keep them up to date, which represents a substantial workload. As a rule of thumb, maintaining software costs at least as much as its initial development, if not twice as much.

Paradoxically, it is precisely the tests least suited to generative AI that must be maintained over the long term. These are exactly the tests for which implicit knowledge is most critical—the knowledge residing in the minds of people who, over time, change projects, switch companies, or retire.

In conclusion: Epic tests require implicit knowledge, which makes “one-click” AI-generated solutions ineffective or inappropriate. This is all the more true given that these tests must subsequently be maintained and, in many cases, automated.

To capture, share, and capitalize on business knowledge (including implicit knowledge) for Epic, Regression, and E2E tests, a visual design is the right approach. And to effectively maintain and automate these tests, a dedicated test IDE is required. It is precisely these two capabilities (among others) that our Yest solution provides.

Yest enables business knowledge to be documented in a graphical way, making it possible to generate the most relevant tests from a business perspective. The graphical approach is highly effective, allowing test design to begin much earlier than usual. There is no need to know the UI details (such as a specific form layout) to understand what information must be provided. Graphical workflows serve as living documentation. When a US changes, they make it possible to quickly identify the affected test steps, especially since different User Stories can be referenced directly within the workflow.

With Yest, it is possible to generate the minimum number of test scenarios required to ensure full coverage of the aspects to be tested. Yest also supports automation through a keyword-driven approach. But it is more than just a test generator based on graphical paths. With its many accelerators, Yest is a true IDE for testers.

Example: If a new US is integrated into an existing workflow, it is enough to update the visual path. Yest will identify the affected test scenarios and suggest automatic corrections. These changes are made transparently, unlike with generative AI, which often behaves more like a black box.

The benefits of Yest are measurable: smarter tests, created 40% faster.

Since version 4.0, Yest has integrated AI capabilities to refactor and standardize existing or legacy test assets (those not originally designed in Yest). This means that if you want to improve your existing tests and create new ones while leveraging visual design, Yest’s AI will help you clean up existing tests, parameterize them, and ultimately translate them into graphical workflows.

But the future looks even more promising. In the next version, Yest will include an AI agent acting as a copilot. It will then be possible to create workflows and tests using chat-based instructions with the AI.

Example: let’s take the new User Story mentioned earlier. With AI in Yest, you’ll be able to ask the copilot where to integrate it into the existing workflow and have it add the corresponding test steps, while Yest automatically updates the underlying scenarios.

One point is particularly important to us: we firmly believe that testers must retain ultimate control over all modifications.

AI should not behave like a black box; the logic behind its actions must be visible and explicit. “Big bang” modifications are difficult to verify and require significant validation effort. That’s why, at Smartesting, we favor smaller, highly traceable steps, in a spirit of true collaboration with generative AI.

There is no doubt that test design is a particularly well-suited use case for AI. However, behind these two simple letters “AI” now lies a wide variety of solutions, just as the word “test” encompasses many different types of tests with distinct nature and objectives. It is up to QA teams to implement the AI solution best suited to each specific testing need.

For Epic tests, E2E tests, and Regression tests, which are highly complex and rely heavily on deep business knowledge, the work entrusted to AI must be closely guided by humans and adhere to best practices. Yest provides this methodological foundation needed to ensure that AI is used to its full potential.