Strategies for selecting and prioritizing smoke test cases

25.06.24

Cristiano Caetano

Introduction
Key characteristics of smoke testing
Strategies for selecting and prioritizing smoke test cases
Gravity helps achieve higher quality through precise test prioritization

Introduction

Before deep diving into selecting and prioritizing smoke test cases, let’s introduce the concept of Smoke Testing. Smoke testing is a preliminary level of testing that is performed on an initial build of the software to ensure that the most critical functionalities work correctly.

Interestingly, the term “smoke testing” comes from the hardware testing practice of turning on a new piece of hardware for the first time and considering it a success if it does not catch fire or smoke.

It is often referred to as “Build Verification Testing” or “Surface Level Testing” because it focuses on ensuring that the core functionalities of the software are working correctly before more in-depth testing is conducted.

The objective of smoke testing is to verify the “stability” of the system to proceed with more rigorous testing types. It is not meant to be exhaustive but rather to quickly determine whether an application is suitable for further, more detailed testing.

Key characteristics of smoke testing

How often should smoke testing be performed?

Smoke tests should be designed to be quick, typically taking a few minutes to an hour to complete. This allows for frequent execution without significantly slowing down the development process.

Here are some recommended times to perform smoke testing:

After each new build: Ideally, smoke testing should be performed after each new build is created. This ensures that the basic functionality of the software is working before more comprehensive testing is done.
Before code commits: It’s recommended to run a simple smoke test before committing code to a repository. This helps prevent introducing major issues into the shared codebase.
After code deployment: Smoke testing should be conducted after the code has been deployed to a new environment. This ensures that the basic functionality is still intact after the deployment process.
Before intensive testing: Smoke tests should be run before starting an intensive run of other tests. This verifies that the build is stable enough for more detailed testing phases.
During continuous integration: In modern development practices, smoke tests are often integrated into continuous integration (CI) pipelines. It allows them to be run automatically with each code change or build.

Typical SMoke Test - prioritizing smoke test cases

Should smoke testing be automated or manual?

While smoke testing is commonly carried out manually, its effectiveness can be significantly enhanced by automating the process to shorten feedback loops. This is particularly important when smoke testing is integrated into continuous integration (CI) pipelines.

Here are the key points regarding the automation of smoke testing:

Speed and efficiency: Automated smoke tests are significantly faster than manual ones. Once set up, they can be run with a single click, delivering results in minutes rather than hours.
Consistency and reliability: Automation removes the variability and potential errors inherent in manual testing. Each test is executed with precision every time, ensuring consistent results.
Frequent execution: Automated smoke tests can be seamlessly integrated into continuous integration (CI) pipelines. It allows them to run automatically with every new build or code change.
Resource optimization: Automating repetitive smoke tests allows QA teams to focus on more complex and exploratory testing tasks. That enhances overall productivity and resource utilization.
Immediate feedback: Automated smoke tests provide immediate feedback on the build’s quality. That enables the quick identification and resolution of major issues before they impact further testing or production environments.

Strategies for selecting and prioritizing smoke test cases

Unlike other testing approaches such as Regression testing or Functional testing, smoke testing typically does not require repeatedly running every test case.

Instead, it involves the strategic selection of test cases based on their risk and priority to quickly uncover critical issues.

As applications increase in complexity, there is a corresponding rise in the number of smoke tests that need to be executed.

This leads to extended execution times. Occasionally spanning hours or even days, which defeats the primary purpose of smoke testing to promptly identify critical issues.

This presents a unique challenge for testing teams: the necessity to strategically select and prioritize tests due to resource constraints and time limitations.

The team is then confronted with the critical task of identifying and focusing on high-priority smoke tests. They need to employ some type of risk-based testing strategies, and continually maintain and update the test suite to ensure relevance and effectiveness.

In the list below, you’ll discover various methods to prioritize smoke tests. All intended to strike a balance between comprehensive coverage and reducing execution time:

Gravity helps achieve higher quality through precise test prioritization

Gravity is a unified platform designed to help testing teams monitor and leverage insights from both production and testing environments. It improves the efficiency of smoke test prioritization.

Its primary function is to produce “Quality Intelligence” by processing the ingested data through machine learning algorithms and Generative AI.

This involves translating raw data into meaningful insights using techniques such as pattern recognition, trend and correlation analysis, anomaly and outlier detection, and more.

Gravity generates data-driven insights for the testing team by comparing real user interaction paths in live production with the tests conducted in testing environments.

These insights enable testing teams to spot gaps in coverage, identify features that are either over-tested or under-tested, and recognize redundant testing efforts in less critical areas.

These insights assist in selecting and prioritizing smoke test cases, ensuring focused test coverage in critical areas, thereby enhancing the effectiveness of smoke testing without relying on assumptions or guesswork.