When we talk about test execution, we often think of test design and execution. Indeed, test design is an essential step if we want to guarantee high-quality execution. We need to identify the test cases, the data, and much more, to ensure that we can then accurately verify the system under test during the execution phase. Neglecting test design tends to lead to lower-quality execution. By quality, we mean execution that strictly follows the test steps and carefully verifies the expected results. How can the quality of execution be measured? Through execution proofs. At each stage, the tester can note their observations, specifically in cases where the results do not match the expected outcome, but also when the expected behavior is observed.

In practice, this activity is often neglected for various reasons, the main one being lack of time. Testers prefer to execute a larger volume of tests to cover a wider scope rather than provide an exhaustive description of the execution results. And even if a policy is in place, the quality of the proof will depend on each individual and context, leading to relatively heterogeneous execution proofs. However, execution proofs are crucial to ensure the reproducibility of the test and the documentation of any defects encountered. Not to mention that in the most sensitive areas, these proofs are simply mandatory and are part of the standards that must be respected. 

Execution proof is, at a minimum, a free text that describes the result of an execution. Its format and richness depend on two main elements: the tool in which the proof is collected and its author. In terms of tools, we generally find the same elements. At a minimum, we find the overall execution status of the test and its test steps. Then each tool is more or less rich: the ability to add a comment, a verdict, an attachment, attach a defect, link a ticket/requirement, etc. Often, completing execution proofs is equivalent to freely noting the observed result. The fact that this is a text field means that comments can lack structure and detail. This makes it difficult to evaluate this evidence over time.

We have briefly touched on the problems related to the lack of standardization. We have pointed out that it can be difficult to evaluate test results over time. And not being able to evaluate a result can lead directly to a loss of confidence in the test results. Without proof, there can be no confidence. And what is the value of an execution if we only know its execution status? Obviously, this statement must be tempered according to the criticality of the tests performed. Less effort is put into evidence for non-critical scenarios than for scenarios subject to audits and certifications. In these cases, the lack of standardization poses a significant problem, as audits and certifications are compromised.

Apart from audits, there is also a more common problem: test reruns. Without standardized proof, it is difficult to compare the results of successive executions. When in doubt, the person running the test will waste time trying to recreate the context rather than progressing with the test execution.

As you will have understood, standardizing execution proofs is a real challenge for quality. It allows you to: 

Increase readability: to facilitate test reruns and analysis. One of the first objectives will be to reduce the workload associated with reviewing test execution. Standardized evidence allows any team member to understand the execution context, even several months later. And this is crucial for test reruns.

Consolidate traceability : standardizing evidence makes it possible to systematize the addition of data and links to requirements or user stories. This makes it possible to guarantee that the execution has been carried out with a focus on covering business rules.

Going further, standardization can even lead to the definition of sets of metrics that can then be aggregated. In this case, it will eventually be possible to extract execution trends (risk, type of errors, recurrences, etc.).

Finally, it is also important to highlight that standardizing execution evidence brings additional maturity to the testing process. It allows the test result to be substantiated (beyond a simple pass/fail) by providing a set of elements that justify this result. 

As mentioned above, the elements to be standardized depend on the features of the tool used to perform the tests. However, a minimum structure can be adopted:

To go further, a pattern can be defined, describing in particular the details to be included in the proof, additional elements such as the environment, the date and time, the profile used, the requirement covered, or any other elements that add value to the proof.

Focus on Xray : 

If you use Xray, you can establish evidence at different levels. 

First, at the test level itself, by completing the “Findings” section, which gives you an overview of the test’s defects and evidence. You can also add an overall comment where you can summarize the evidences and make an overall assertion about the test in order to provide visibility and avoid having to go into detail for each step.

The second level is found per step, where you will find the concepts of defects, evidences, comments (in the Jira sense), and actual results. The Actual Result field will therefore be the most suitable for writing proof, as it can be structured. You can choose to structure your proof in a table, detailing the observed result. In addition, adding screenshots of the actions performed will facilitate the review phase. 

You understand the challenges of standardizing execution proofs and now want to implement it operationally. There are various options available to you for implementing proof standardization.

If this activity is carried out manually by a human, you must first define the scope over which these proofs will be implemented. First, which projects and which tests are concerned, at least as a priority. It may be wise to start with the most critical environments, applications, and tests before systematizing the addition of proof. Next, define the expected content of the proof by test type. You can refer to the elements contained in the proof presented above and decide what is most relevant to your context. You can also choose whether all steps should have evidence. Ultimately, yes, but to implement the approach iteratively and incrementally, you can already choose to standardize proofs on the test steps in errors. And gradually increase and refine the scope of proof standardization.

Another option for implementing proof standardization is to use an AI agent.

Our Lynqa AI agent automatically generates proof for each test step executed. Whether the test step is successful or not, Lynqa describes the observed result through an assertion and establishes a status for the test step. Having this activity performed by an AI agent guarantees the uniformity of the proof, respecting a precise structure. 

In addition, entrusting this activity to an AI agent reduces subjectivity and limits interpretations specific to human analysis. It also reduces the human effort required to perform tests. Although human review of the evidence provided is still necessary, this process is faster than a completely manual one.  

In summary, an AI agent is an effective lever for both initiating a process of proof standardization and lightening the workload of existing teams. This will strengthen confidence in the product, facilitate test review, and take full advantage of the benefits offered by the standardization of execution proofs.