Test Automation has been a part of software testing and quality for a good 25 years now – probably longer if you include some low-level whizzes who were ahead of their time. However, it is now firmly on the radar of almost all organisations – from enterprise to start-ups – to some degree. The apparent silver bullet has been widely recognised and the virtues of which, are now being discussed at the highest levels.
As a result of this, it has now become a breeding ground for over-hyped expectations, supported by underwhelming budgets. As a company who provide services to clients in this space, this is particularly dangerous and one we tread with caution. We aim to right the expectations and help clients maximise their investment.
Test Automation is a great building block to faster delivery, reduced risk and increased productivity. There is no doubt that all these can be achieved with the right sponsorship, the right approach and the right investment. The question is: “Why do organisations seldom provide these three things if the outcome is so positive to the business?”
I think the C-Suite and Senior IT Leaders need to consider 5 key things that can help ensure they make Test Automation successful:
Test Automation should be complementary to the work your development, test and release teams do (you could even use it within production if you are brave). It should be a productivity enhancer to every one of them – a way to increase their throughput daily. One of the myths of Test Automation is that it there is an “us” and “them” – testers versus engineers. This is not true. The value of a great engineer can never be in question – they can do the coding of the frameworks, understand the complex API’s, but more importantly, they should be coaching everyone in the teams to be more productive.
There is a skills shortage and the greatest asset you have is your people – upskill them, set their goals higher. It can be a catalyst for retention and recruitment and at the same time increase your speed of delivery.
One of the biggest red-flags we see when we start to work with organisations is that they refer to the SLA’s that they have in place with their larger suppliers – usually around volumes of test, % coverage and regression packs run etc. The truth is that these are not totally relevant.
The relevance is in the “value” that the automation brings to your release, business and team. The value is determined by the level of risk you are mitigating against. It’s about ensuring there is enough time for your exploratory and functional testing teams to spend on the areas that need that inquisitive validation to investigate and explore – assessing business risk. It’s about ensuring your developers are picking things up early and that coding standards are high. SLA’s are designed as contractual targets and to show pretty graphs, they are generally not used to measure against business risk – the only reason why you are testing in the first place!
Look at these carefully and make sure you are comfortable in the value that you are getting from your investment.
Having worked in testing for over 20 years, there is no doubt testing has become more prominent in discussion with Senior IT leaders. Its on the agenda – and that’s great. However, a key issue is that Test Automation, in my view, is budgeted for incorrectly and its one that can be easily fixed. It is generally seen as part of the traditional testing spend, and it therefore fits within the level of investment that testing usually does – 20-25% of project spend.
However, where I have seen automation work incredibly well, is when organisations invest in it like a development project. In truth, the investment must match the ambition – in one circumstance we have been involved in, the result was game changing – monthly releases went to daily releases across 7 brands in 18 months. The initial investment was based on a move to CI/CD set-up and then to mobilise and integrate that for every product owner/brand.
The subtlety of this may surprise people, but if you have the mindset of this as a project, particularly if you are looking at legacy application / regression builds, then the approach will get more buy-in, more budget and deliver better results.
It is also worth speaking to your CFO and see if you can add “Regression Packs” to your Balance Sheet as an Asset – its more complex than you think, but as a reusable asset it can mean different financial accounting that makes the investment more attractive. It’s certainly worth exploring.
When a project is allocated its funds to deliver, the project manager is nearly always focussed on the immediate delivery milestones – getting the solution over the line. They are less focussed on the true lifecycle of the application – i.e. managing the true risk and cost to the business over time. As such, they allocate their budgets accordingly – they spend the money fixing today’s issues – whether that be in development, test, environments etc. That is expected and understandable.
However, in this approach, the need for a sound regression pack, a well-thought-out automation suite is not at the top of their agenda. The cost incurred now, will only be realised in a BAU context when they have moved on from the project. The value to the business is critical, but the immediate value to the project manager is limited. It’s a constraint that is self-inflicted by the set-up of projects and the associated finances.
Although a lot of organisations are moving towards Agile / Dev Ops this point may be less relevant, but as there are still so many organisations who are doing iterative development or dealing with legacy applications in a project world, then this is still a major blocker to Test Automation. I have seen examples of an “Automation Tax” being introduced, but the language can be emotive – however the principle remains – a set allocation of budget/finance dedicated to building a reusable asset.
One of the things ROQ often gets asked to review is the Test Automation strategy of an organisation. The main driver for the conversation is usually that an organisation has pockets of automation in projects, sprint teams, third parties etc. There is a plethora of tools being used – some open source, some paid-for. There are mini-successes and there is a lot of shelf-ware.
Whilst experimentation and the idea of innovation is great, the truth of the matter is that most of these things fail because its not part of a co-ordinated plan. It’s not that the tools are wrong or that the intentions are negative, it’s because there is no North Star. There is very little mapping of the technical landscape before tools are chosen. There is seldom a strategic view considered of the technology or platforms going forward.
All these things should feed into the test automation approach and therefore it will help you leverage the skills you have internally and remove wasted effort on unfruitful experiments.
I have picked these 5 as they are the most common ones I see – and they are all fixable. There are lots of opportunities to embed Test Automation into your delivery capability and if done well they can really drive the business forward. I referred earlier to the silver bullet, but with increased pressure on technology functions to deliver more at an even faster pace, then Test Automation is the only way to make this a reality. It’s worth thinking about it seriously and investing in it appropriately.
Stephen Johnson - Co-founder & Director, ROQ