Since 2020, we’ve worked with hundreds of startups transitioning from manual software testing to test automation. Over that time, we’ve repeatedly seen teams underestimate the costs of test automation maintenance.
Specifically, teams don’t anticipate how much time they’ll have to spend on maintenance, and how disruptive it’ll be to their productivity and release pipelines.
To quantify these costs, we surveyed 77 startup software teams in the U.S. and Canada about their experiences with test automation. We’ll share the results in this post.
But first…
A quick primer on test maintenance
When you push a code change to your app, it can break your tests.
For example, if you change your “Sign in” buttons to “Log in” buttons and some of your tests are still looking for “Sign In” buttons, then those tests will fail. These broken tests present “false positive” failures, because the failures reflect a problem with the tests, not a problem with your application. These failures can result from any sort of change to your application that isn’t represented in your tests.
Test maintenance is the work done to update automated tests to reflect the latest, intended version of your app. The more you update your app, the more someone needs to investigate and then update your failing tests.
If you don’t perform regular maintenance, your automated test suite will quickly fall out of date and won’t effectively do its job — finding bugs and other issues in your app.
As the data will show, test maintenance can represent a time-consuming chunk of your automation efforts, and — between finding the root cause of test failures and then updating test automation scripts — it’s no-one’s idea of fun. It can take your engineers away from shipping code, and it frequently bottlenecks release pipelines.
The data: How much time does test maintenance actually take?
Creating and maintaining tests in open-source automation frameworks like Selenium, Cypress, and Playwright can be a drag on agile development teams aiming to ship fast and frequently. (Selenium is notoriously painful to maintain.) They all require someone with technical skills — like a developer or QA engineer — to dig around in code to find the offending part(s) of the test to fix.
In fact, software developers often deliberately neglect to update their end-to-end (E2E) automated test scripts in favor of delivering more code. After all, most devs are incentivized to ship, and software teams have to ruthlessly prioritize their time and resources.
No- and low-code test automation tools can make maintenance a bit easier, but they still carry maintenance costs. (More on that, shortly.) Our survey respondents mostly use open-source frameworks, with a handful using no- or low-code tools.
In our survey, 36 percent of startup software teams admitted to failing to keep their automated tests up to date due to the time commitment involved.
It’s not that these teams got back a lot of time by skipping maintenance altogether — they still spent hours every week writing and updating tests, but nonetheless fell short of up-to-date test coverage. Despite their investments, their test suites can’t reliably give them confidence in the quality of their respective apps.
Of the teams who do keep their automated test suites updated:
- Over half of the teams spend at least 11-20 hours per week on test maintenance
- About a quarter of them spend at least 31 hours per week — approaching the equivalent of a full-time hire.
The more complex your app becomes, the more testing effort — including test maintenance — is needed. Indeed, there’s a clear correlation between the number of engineers working on your app (which is a proxy for app complexity) and the amount of time spent maintaining automated tests.
That is, as your team and your app grow, so will your maintenance costs.
| Engineers working on the app | Hours per week maintaining tests (median response) |
| 1-10 | 6-20 |
| 11-20 | 11-20 |
| 21-30 | 21-30 |
| 31-40 | 21-30 |
| 41-50 | 21-40+ |
What about using no-/low-code tools?
Compared to working with open-source frameworks, it tends to require less time to write and maintain tests in no- and low-code tools.
But less than “a lot” is still a notable amount.
If anything, the tendency to underestimate the costs of test maintenance is worse when a software team is adopting a no- or low-code solution. They assume “no code” means “easy,” and that, correspondingly, maintenance will be easy, too.
We saw this all the time over the several years we sold a no-code test automation tool (before we enhanced it with AI and optional test automation services). We even have customers who signed on for our test automation services because they couldn’t stay on top of maintenance with an easy-to-use, no-code tool.
It’s also important to note that not all no- and low-code tools are created equal — some are much more complex than others, making them more difficult to learn and use.
According to our customers, TestComplete and Katalon are among the most complex, and Testim is one of the easier ones.
How to avoid the costs of test maintenance
Clearly there are significant time costs involved in test automation maintenance, and if you don’t have a QA team, there are other ways you’d rather your engineers spend their time and attention.
So here are the ways you can automate your end-to-end tests while keeping maintenance to an absolute minimum.
1. Use a test automation tool that makes maintenance easier
As we’ve established, test maintenance is generally less painful in no-/low-code tools than it is in open-source frameworks.
But “no-code” isn’t the same as “intuitive,” so make sure you’re choosing a solution that’s as easy as possible for your team to learn and use.
We’ve designed Rainforest to be the most intuitive no-code test automation solution on the market. Anyone from your team can automate, interpret, and maintain automated tests without any training.
Rainforest QA is an all-in-one test automation platform, so it includes everything you need for the testing workflow, including an API and CLI to integrate tests into your CI/CD pipeline.
2. Let AI handle maintenance for you
There are several ways in which AI-powered testing in Rainforest QA helps you avoid unnecessary maintenance efforts and increase your release velocity.
Reliably identifying elements in your app
In most automated testing tools and frameworks, a slight change to an element in your app — like a button or form field — can break the relevant tests. Whether the element’s visual appearance or naming convention in the behind-the-scenes DOM code has changed, a test relying on a single indentifier (or “locator”) for the element can fail when it can no longer find a match for that identifier. (Automated tests that frequently fail due to minor changes like these even have a name: “flaky tests.”)
Tests in Rainforest are more robust because they rely on three different types of identifiers to locate elements in your web application. These include visual appearance, an automatically-identified DOM locator, and an element description automatically generated by our AI. A change in any one of these identifiers won’t break your test.
Automatically updating tests
When you make intended changes to your web app’s appearance or functionality, Rainforest’s AI can often automatically update — or “heal” — the relevant test steps so you don’t have to.
Get a quick overview of the AI’s self-healing capabilities in this 45-second video (or check out an in-depth, 8-minute demo):
Determining the app changes that matter
Rainforest’s visual validation algorithm has been trained via machine learning to simulate the judgment of a human tester — if a visual change in your app’s user interface is so minor that a human tester wouldn’t notice or care, Rainforest will allow it to pass.
So, inconsequential visual changes to your app won’t trigger false-positive test failures.
3. Write fewer, shorter test scripts
This is a core component of the testing best practices and methodologies we recommend to our customers.
Fewer tests
Designing automated test coverage is largely about prioritization — applying the Snowplow Strategy, focus your automated regression tests on the most critical user paths and functionalities in your app.
Once tests cover the most important flows, you’ll see diminishing returns on additional tests. At some point, you’re adding future, recurring maintenance costs in return for having confidence in functionalities that aren’t all that important. As part of your testing process, recognize when that tradeoff isn’t worthwhile.
Some rules of thumb:
- Don’t waste time testing functionalities that you wouldn’t fix right away if they broke.
- Treat your test coverage priorities like a product roadmap: “nice to haves” go in the backlog.
Shorter tests
The longer a test is, the more points of potential (false-positive) failure it presents.
So, make each test script as long as necessary and as short as possible. You’ll enjoy faster test executions, an easier time debugging tests that fail, and ultimately, lower maintenance efforts.
4. Don’t use E2E test automation for new features still in active development
Unit testing and integration testing are useful (and sensible!) early in the software development life cycle. But it doesn’t make sense to create automated E2E tests for a new feature until it’s largely stable.
Otherwise, you’ll spend a disproportionate amount of time updating your tests to keep up with the feature. For these test cases, the flexibility of manual testing makes it a better choice.
5. Look for opportunities to re-use test snippets
Most test automation frameworks and tools will offer a way to create “snippets” of test scripts that you can re-use across different tests. You can update a snippet once, and the change will replicate across all the tests where the snippet is embedded.
In Rainforest, we call these “reusable snippets,” and they’re helpful for cutting down the time needed to create new tests and maintain existing ones. Common test cases for them include testing login and account creation flows.
Without these snippets, if you updated your login sequence, for example, you’d have to update every automated test case that covers your login sequence. With snippets, you only have to make an update in one place and it gets applied across the board.
As you’re writing your tests, be mindful of flows that tend to repeat — these will be good opportunities to save time with reusable snippets.
6. Make debugging easier with detailed test results
Part of the pain of the test maintenance process starts with debugging — figuring out what needs to be fixed.
Detailed test results will give you enough information to quickly figure out where and how each test is failing, and even suggest why it’s failing.
Whatever test automation solution you use, we recommend — at a minimum — making sure you have access to the types of test result details we provide in Rainforest, including:
- Steps to reproduce failures
- Video recordings of tests
- Browser and HTTP logs
7. Use a test automation service
If you’d like your team to avoid the costs of test maintenance altogether, Rainforest QA’s test automation as a service will handle 100 percent of automated test creation and maintenance so your team can focus on higher-value work.
As part of the service, we’ll dedicate one or more experienced Test Managers to your account. Simply tell them what you want tested — type a message or record a video — and they’ll take care of the rest.
- They’ll embed in your workflow and comms tools (e.g., Linear, Jira, Slack, Teams), deeply learning your product and priorities.
- They work in or near your time zone and speak fluent English, so communication is easy and efficient.
- They’ve all worked with us and our platform as quality assurance professionals since at least 2017 and undergo regular training and evaluations.
- They’re all consistently highly rated by customers — otherwise they wouldn’t still be with us.
The Test Managers also proactively review your test failures, updating the tests that need it and sending you the legitimate bugs.
Plus, they work within Rainforest’s no-code, AI-accelerated platform, so they’ll be able to create and maintain your automated tests as quickly as possible.