12 months Opinion Pieces

What is Optimal Test Coverage for Test Automation?

So… test automation coverage. Coming at this topic as a test engineer, you might expect me to say, “automate everything!”. It is no secret that automation is great because it helps you deliver high quality, quicker and with less cost in the long run. But do not worry, this is not going to be one of those blog posts where I rant about why automation is the bees-knees, and why you should all be automating everything.

Automation is what I do, and I have been doing it every day for the last 3 and a half years. In that time, I have really seen the value it has provided to our clients, but I have also seen occasions where we have inherited automated regression packs from incumbent suppliers that have never delivered the value that they should have. The reason for this varies, but it is usually down to test packs that do not get updated frequently, or do not provide the coverage that they are expected to. So, what should we do to gain optimum test coverage, without sacrificing time? Here are three areas that I think will be of interest:

Focus on automating high value applications with frequent releases

In terms of which applications you should automate, you are likely to see the most value when automation is delivered against an application that is frequently updated. These are the applications that can quickly become a drain on testing resources, with frequent releases and a demand to run manual regression tests, and to test new features for every release.

It is also the easiest type of automation to get budget for because it is the easiest one to demonstrate the money it will save the business. Depending on how frequently your application is released, you could see a return on your investment in a matter of months, or even weeks.

Risk-based approach

As with most testing, you would likely want to align your automation delivery based on risk. It is pointless spending a few weeks automating the testing of an application if it is used by only a handful of people for non-business critical functions. Whereas an application used by the entire business for critical actions should be high on your list of applications to automate.

Once you have selected an application that you want to automate, then the next question arises. Which tests should I automate? Realistically, as much as you may aim for 100% automation coverage, it is unlikely to happen. There are some tests that are just too technically challenging, they would take too long to script for the ROI to be seen in a realistic amount of time. In my experience, aiming for 60%+ is aspirational for most organisations who are taking test automation seriously.

However, you should take a risk-based approach. Any tests that cover critical pieces of functionality should be your first target for automation, and then work down the list until you are left with your lowest risk tests. Sometimes budget can run out before you finish scripting all the tests, but with this approach you have automated your highest risk tests and so you are left with an automation pack that delivers as much value as it can.

Target problem areas

I have also worked with some clients who prefer to automate areas where they frequently see issues, even if that area is not the highest priority. Say you have an application with a complex piece of functionality, that frequently has issues but is not used as much as other parts of your application. I would target that area with automated tests on the basis that it is the most likely area you could see issues.

People also overlook the API layer but I would argue that this is the most important area to target because you will find defects which are the root cause of issues usually found later on. Gaining better coverage at this level means that you will find fewer issues later in the development lifecycle when they are a problem to fix.  

In summary, test automation can be a juggling act. As the automation engineer, you want to automate everything, because nearly everything can be automated with a varying degree of complexity. The business probably wants it done within a specific timeframe or budget. The application has its own specific needs around which areas should be automated. It is down to you to weigh up the priorities to deliver an automation solution that provides the most value, taking a risk-based approach.

Graham Hurlston – Senior Test Engineer @ ROQ


Comments are closed.