Smoke Tests: What is-it?

Smoke tests are, according to ISTQB definition: 

A test suite that covers the main functionality of a component or system to determine whether it works properly before planned testing begins.

You’ve probably come across them before, and certainly under a different name, because they’re so important that they’re entitled to many names depending on the team. The most common terms are sanity test, intake test, and confidence test.

Whatever they’re called, there’s a kind of urgency or “vital” minimum that they’re supposed to provide.

The term “smoke” would come from testing printed circuit board connections. They were checked to make sure they didn’t start “smoking”, to ensure that the circuit hadn’t been fried when powered up. If smoke was present, the circuit was “dead” due to a fatal fault, and there was no point in testing it.

The idea behind “sanity” testing is the same, as is the term “confidence”. Which seeks to ensure that the software received is of sufficient quality to be accepted. We don’t accept a package that arrives destroyed!

In practical terms, smoke tests are an initial verification of the quality of the software delivered. This initial verification must ensure the absence of “critical” bugs. By “critical” bugs, I mean bugs that make the whole software (or a significant part of it) unusable.

These tests must be carried out each time the software is re-delivered. And it is therefore essential that they are carried out quickly… otherwise, a great deal of time will be lost, if not more!

As you can see, a smoke test campaign must meet 2 imperatives: speed of execution and usability of the basic functions of the software under test.

A tester needs to be able to determine this “vital minimum” of tests to meet the software’s usability requirement while using as little time as possible.

To achieve this, the tester needs to be able to identify the software’s core functionality. By core functionality, I mean a fairly limited number. I should refer to them as Macro-functionalities. If we work in Agile, these macro-functions could be “Epics” or “Features”.

Once these functionalities have been identified, it’s time to determine whether there is a “vital” interest in making each of them usable. In almost all cases, the answer is “yes”, but it’s always worth asking. For a mobile mail application, for example, it’s worth asking whether it’s worth having a smoke test for managing messaging options or folders.

Once we’ve made our selection of macro-functions, it’s essential to identify the key paths for each of these functions. These are also known as “nominal” paths. In concrete terms, these are the simplest and most commonly used paths.

When this selection is complete, we have the basis for our smoke test campaign.

Please note that in this paragraph I have only talked about functional system tests. It can be “vital” to integrate non-functional tests, and it’s important to integrate lower-level tests. Unit tests (generally component tests) and API tests (generally integration tests) are very fast tests that cover the software well. It’s a considerable asset for smoke tests.

Smoke tests must :

They are a perfect target for automation. Ideally, they should be automated or be the first tests to be automated. What’s more, automating these tests enables you to achieve interesting method coverage with automated tests. It’s often more interesting than having only one or 2 highly automated functionalities and the others not.

It is typically very interesting to take this automation a step further, by integrating all smoke tests (including tester tests, which are the main focus of the article, and lower test levels (in the ISTQB sense)) into the CI/CD chain.

Smoke tests are essential! They save a lot of time, especially as they’re a practical application of early testing. 

But running an effective smoke test campaign is no easy task. Apart from the fact that component and integration tests must be included, the choice of system tests is more complex than it seems. The tester and his team must pay particular attention to the selection of these tests and, where possible, to their automation… while remembering that these tests are not set in stone and that the campaign must live and evolve with the software under test.

Read more about testing on La Taverne du Testeur (French)