Regression Test Coverage - Your Product’s Quality Depends on It

Regression Test Coverage - Your Product’s Quality Depends on It

Leo C

Leo C.

7 minutes

Nov 26, 2021

Link copied!

Regression Test Coverage - Your Product’s Quality Depends on It

One of the most used programming jokes is: „While there is a code, there is a bug“. Although experienced developers can avoid a lot of them, there will hardly ever exist software without any bugs during the production time. A software bug is a term that includes any kind of error, flaw or fault in a system or program that causes incorrect results or unintended behaviours.

Every change is a chance for a potential bug

We can all agree on one rule – the most important thing is that the bugs are detected and resolved before the project delivery. Some of them are minor, harmless and don't take long to resolve, while others may cause software or application to crash entirely or, even worse – to lose important data.

One of the biggest problems in development is that any newly introduced code can cause a bug by changing or breaking already existing software features. Every single change is a chance for a potential bug – which includes changes in settings, configuration, database or even permissions, and often even the most experienced developers can miss the bugs during the production.

So, what should you do to prevent this from happening?

The best way to prevent problems with introducing new code is to perform regression tests. What exactly are they? Regression tests are a series of tests designed to check if the newly introduced code and features changed or in any way broke the existing features of the software. Sometimes even fixing one bug can unintentionally create another – so regression testing needs to be executed to ensure the fix didn’t cause any abruption.

Sadly, most software products aren't covered by regression tests, which puts them at significant risk of critical bugs occurrence during production. Although creating high-quality and reliable regression tests takes time and money, they are always worth it – because they make sure income-generating functionalities are up and running 24/7. This process is absolutely necessary to ensure that any code changes won't affect the existing features of the project.

Manual or automated regression testing?

There are two types of regression testing: manual and automated. The manual process includes, as the word implies, manually revising all of the existing features. On the other hand, the automated regression tests, which are the focus of this article, make your job a lot easier.

Automated regression testing is a set of tests written in a form of scenarios that are designed to follow a certain path in the application. The tests „go through“ the entire application and click certain buttons and links, creating a path – and unlike with manual testing, the computer does all the work for you.


A crucial part of software development

One of the common misconceptions is that you don't need automated regression test coverage if you already have Unit tests (a method that tests individual units of source code to determine whether they are fit for use). Being a great addition to unit tests, regression tests are run daily and weekly – in Devōt we recommend running them before each release.

We prefer the Cucumber automation regression tests – because it provides better collaboration. It provides better collaboration with non-tech-savvy people involved, as they usually won't understand the details of the Unit test, but they understand an example written in Gherkin syntax. The Cucumber is a software testing tool that consists of feature file and steps definitions – by using special keywords that give structure and meaning to the steps in the scenarios that are being executed. Step definitions connect Gherkin with the code and elements of the app.

Regression test coverage – what is it all about?

The most crucial part of performing successful regression tests is to ensure proper test coverage. If the coverage is insufficient, the tests won't result in discovering critical bugs before reaching production and potentially causing problems.

What is test coverage anyway, and why is it so important? Imagine all of the features existing in a certain application – test coverage, as the word implies, covers all possible flows and „happy“ paths with test scenarios. We focus on those „happy paths“ (the path and flow that a typical user goes through on the website), in order to cover and secure those paths. For example, if you sell something on your website, you need to have test coverage over every possible flow that leads to the sale.

If the introduction of a new feature causes a „break“ in the existing features on your page, the regression tests report will let you know – and prevent the website or app malfunction, as well as ensure your business continuity.

Regression testing is not the same as retesting

The most important thing to test in this process is the „happy paths“ and business-critical user flows, but what about the edge cases? Well, automated tests take some time since they simulate a user going through all the critical flows for the application to work. Edge cases are usually tested with unit tests, which are much faster in checking code functionality, and they are usually done before the final testing with our regression test scenarios. More coverage means more time and money, but it prevents bugs and crashes from happening.

Later in the article, we will show you how we structure a test coverage on the example of the Contact us form on the Devōt website.

Some companies believe that regression testing is just retesting. However, regression testing ensures the systems still work properly once the changes have been added – hence the implementation of regression testing has a much broader grasp than pure retesting.

A valuable regression test coverage offers developers a proper analysis of the main issues that are often caused by introducing new code into existing programs, as well as enables evaluation of the quality and functionality of newly written code. At the same time, it ensures that the new code doesn't break any existing features, and help to avoid any critical bugs in the production. The syntax is usually pretty simple and straightforward – it is adapted to understanding the features step-by-step.

Excellent, but how do I run regression tests?

Webdriver launches a browser while executing a test case – it can serve as an excellent presentation of the software with simple syntax, understandable to the layman. We can follow the steps, and actions that are happening (or should be happening) on the page are being executed in our command-line interface. We can run the whole repository of our test scenarios or focus on some specific critical flows. Regression tests can be run locally on your computer or on a specific server, like Jenkins.

Why is regression testing so important? Because it provides invaluable insight into issues that can occur every time you introduce a new line of code into your software. The code is often imperfect on the initial release, and it is quite usual to make improvements over time. That doesn't mean the code is not high quality; it means that there is always room for improvement – so the testing is implemented in every stage of development, allowing the system to keep evolving. Regression testing is by far the first and the best line of defence against risk – it makes sure the new code makes the system better, not worse.

In Devōt, we use Cucumber automated regression tests. It is a tool that consists of feature files and steps definitions. Feature files use Gherkin syntax, which uses special keywords to give structure and meaning to the steps in the scenarios that are being executed. Step definitions connect Gherkin with the code and elements of the app.

By mixing Cucumber automated regression testing and unit testing, we are making sure to use all of the advantages of both – we write the examples and path in Gherkin and use unit tests to cover the edge cases. We think of the Cucumber as a collaboration tool, not just a testing tool. Even the creator of Cucumber, Aslak Hellesøy, called it “The world’s most misunderstood collaboration tool”.

But what does it all look like?

We were hoping you would ask that – and the best way to show you is to use our website as an example. So take a look at how the Cucumber automation test would look like for the Contact Us form on the Devōt website.

Contact us to learn how your system can benefit from proper regression test coverage.