2 years Opinion Pieces

Test Automation: The Right Way Up

Taken from Test Magazine September 2017 - Jonny Fletcher, Test Architect, ROQ argues DevOps is driving the adoption of wider test automation.

Across the IT industry, the demand to deliver solutions ever faster whilst satisfying the expectation of improving quality continues to gather pace, as users become ever more sophisticated and demanding. Against this background, the DevOps approach has become a hot topic and is becoming widely adopted as established companies seek not to be left behind by the so called ‘disruptors.

A healthy DevOps workflow allows all members of the IT organisation to collaborate seamlessly. The DevOps adoption has become synonymous with continuous integration/continuous delivery (CI/CD) methods, but whilst they may have common goals and are often used in conjunction, there is a subtle difference. Whilst CI/CD is focused on automating the processes in software delivery, DevOps also focuses on the organisation change to support great collaboration between the many functions involved.

Test Automation is Essential to DevOps Success

With the widespread adoption of agile development methodologies and the need for more frequent deployments, the volume of testing required has increased significantly, especially around regression testing. With this increased volume of development, and the associated testing this brings, comes the need for test automation. Put simply, DevOps cannot succeed if it still requires lots of test cases to be executed manually.

Implementing an effective test automation solution isnt easy and a lot of the hard work comes at the beginning, not only in defining good test cases with sufficient acceptance criteria, but ensuring that your application has been developed with test automation in mind especially true with testing at the user interface (UI) layer.

Establishing a Test Automation Strategy

DevOps requires developers and test engineers to work together to help create test scripts and maximise test coverage. These scripts and code, supported by CI/CD tools, are used to generate builds, deploy them and test them automatically. A more strategic role for test automation is possible where people have the technical and management experience. This architect role means defining a quality strategy aligning with the DevOps culture, which include assisting in defining requirements; creating strategies where 100% automation coverage is not possible; defining quality metrics and measuring and analysing those metrics. In this role, it is more a case of not just finding the bugs but becoming responsible for preventing bugs.

Automation Test Pyramid

When and Where to Automate

In order to be truly effective your test automation efforts need to address at least three different levels across the solution. In non-DevOps software development, many people end up inadvertently falling into the ‘ice cream cone anti-pattern’ for testing by putting more emphasis on automating at the UI level. However, a more practical approach is one that flips that ice cream cone upside down. This approach, the ideal test automation pyramidmade popular by Mike Cohn, improves the ROI of automation and guarantees that you will receive the most benefits from your testing effort.

Devoting effort to automate low level unit tests that prove the functionality works, perhaps using a test driven development (TDD) and/or behaviour driven development (BDD) approach can prove a robust and effective approach. Equally automated tests are also very effective at the integration/API layer, proving that our components (particularly with a microservices design) and external solutions work together as intended. However, as user experience is key for our discerning users and compatibility with more platforms/devices/operating systems is also a challenge, automated tests to be truly comprehensive also need to validate that our solutions behave correctly at the front-end (GUI layer) unless our solution is ‘head-less.’

DevOps Demands of Test Automation

Simply – the automated scripts in this world must provide instant and accurate feedback to support decision making. In the CI/CD context we need to make decisions up-front around what to do in the event of failure, i.e., which failures should prevent deployment or halt the process. Whilst there has also been a requirement for automated scripts to be reliable, the emphasis becomes even more acute when any false negative/failed run means an unwarranted delay to delivery or release. The emphasis on portability can also become more important, as the automated build process may spin up and deploy to an entirely new environment or set of devices and invoke our scripts to run on them.

Establishing Test Automation in the DevOps Context

So how then do you get started when you need to derive the essential benefits of test automation in an emerging DevOps culture? At ROQ we apply our Understand>Define>Deliver approach:


Initially you need to understand key stakeholders, where the organisation is testing capability wise, the technology stack that may require testing, what tools/environments/data provision are already in place, along with what are the key business and technical risks facing you.


Once the background is established you can then define what your approach/strategy for automated testing should be. What skills, environments, data, tools will be required? You should conduct proof of concept exercises for additional tools as necessary.


After this is agreed you can start to on-board resource; develop skills; build frameworks; develop scripts to test all layers; secure test data; build environments; integrate with the orchestration and other tools; and start to deliver the desired benefits from test automation.


In a fast-changing world, which demands more frequent deployments to keep up with the pace of change, test automation can help drive huge efficiencies in a DevOps environment. It is imperative that all parties engage as early in the process to maximise test coverage and to provide feedback to support the decision-making process. However, test automation isnt a silver bullet and doesnt replace the need for great manual testers; it purely enables them to expand the scope of the overall test strategy and to concentrate on exploratory testing and the overall user experience.



Leave a Reply

Your email address will not be published.