If you’re just getting started with software testing, or you’re looking to switch from manual testing to automation, the idea of a free test automation tool can be pretty appealing.
Theoretically, a free tool means you can start producing automated test coverage with little or no financial risk. But that’s not how it shakes out in reality.
Almost all “free” automated testing tools come with (sometimes substantial) hidden costs.
In this post, I’ll highlight the hidden costs of ten of the most popular free test automation tools so you can make an informed decision about the right testing tool for your team.
Open-source test automation frameworks
- Selenium
- Appium
- Cypress
- Playwright
- OpenTest
- Watir
- Capybara
Commercial test automation tools with free plans
- Katalon Studio Free
- Reflect
- Testim
Rainforest QA’s no-code test automation platform avoids most of the costs and challenges of free tools while giving you everything your team needs to get up and running with test automation.
Talk to us about setting up a Rainforest plan that fits your needs.
Open-source test automation frameworks
Open source automation frameworks are indeed free to use (with many caveats, as you’ll read, below).
Plus, they tend to be customizable and come with free “support,” since you can ask for help from the communities behind these projects. Of course, there’s no guarantee the help will be timely or correct.
The costs of open-source test automation frameworks
Expensive technical headcount
The main challenge with open source tools is that each one requires coding skills and familiarity with the framework itself.
That means if you adopt an open-source framework, you’ll need expensive technical personnel to create and maintain your automated test suite. These devops personnel typically represent the largest expense associated with open source automation frameworks.
You can hire a QA engineer, and/or re-allocate some of the time of your software developers away from shipping code. It’s either going to cost you money in additional headcount, or cost you productivity as your developers wrangle with a complex automation framework.
Adding important testing functionality
Open-source frameworks don’t come with everything you need to automate tests. Selenium, for example, simply automates browsers — it doesn’t even include features for test management or insightful test reports.
That’s why there are entire ecosystems of plugins available to extend the functionality of these frameworks. Some of these plugins are also open source (and free), but some of the better ones are commercial.
For instance, none of these frameworks performs automated visual regression testing, which is necessary to have confidence in the user experience of your app. They simply provide functional testing by interacting with the front-end code operating behind the scenes of your web app (i.e., in the “DOM”).
This is significant because, by default, these frameworks don’t test what your users will see; they test what the computer “sees.” For example, a user wouldn’t be able to interact with a button hidden by a popup, but a Selenium test step looking for that button would pass because it found the button’s code in the DOM.
If you want to add visual regression testing to an open-source testing framework, you’ll need to add a plugin. Applitools, a commercial tool, is one of the better plugins to add AI-powered visual testing to many of the tools in this list.
Test maintenance
As your app inevitably evolves in ways both big and small, you’ll need to update (or “maintain”) your automated tests. Otherwise, your tests will incorrectly fail when they can’t find elements in your app that have changed or disappeared.
Test maintenance in open-source frameworks can be a drag on your software development team’s agility. Particularly if you’re a startup, lost productivity means lost money.
First, you have to dig around in code to find the affected lines. Then, being able to fix the affected lines of the test script means knowing the relevant locators to update. Locators are CSS classes, Xpaths, element IDs, or other DOM code that open-source frameworks use to identify various elements in your app.
If you don’t know your app’s locators by heart, you’ll need to dig around in the front-end code of your app to find the relevant ones. It’s not rocket science, but it’s extra, mindless work that adds overhead to the testing process.
Provisioning machines to run tests
You need real or virtual machines to run your tests on.
If you’re running a limited number of tests sequentially, running them on a single local machine might be fine for your purposes. But as your test suite grows, you’ll want to run your tests in parallel so your dev team doesn’t have to wait an hour or more for them to execute.
Running more than one test at a time requires provisioning multiple machines. That means either spending time and money acquiring, configuring, and maintaining machines yourself, or paying for a “test grid” of machines through a provider like BrowserStack or SauceLabs.
Unlike open-source frameworks, Rainforest QA is a fully-codeless test automation tool: you don’t need coding skills to create or maintain automated end-to-end tests, so anyone on your team can contribute to updating your automated test suite. There’s no need to hire any specialized technical headcount.
Plus, Rainforest is an all-in-one test automation solution — you don’t have to add any plugins or provision any machines to run your tests, because it comes with everything you need to write and run automated tests for web applications.
Selenium
Selenium is the oldest open-source framework for automated testing of web apps. It has compatibility with many different programming languages (JavaScript, Groovy, Java, C#, PHP, Python, Perl, and Ruby) and can automate cross-platform tests across all the major browsers (Chrome, Firefox, Edge, Internet Explorer, Opera, and Safari). You can run it on all the major operating systems, including Windows, Linux, Mac OS, and Solaris.
One of the components of Selenium, Selenium IDE, allows you to record your actions in your web app as test steps. It’s a faster way to create automated Selenium tests vs. coding, but it’s limited to creating basic test steps. For test cases involving any sort of complexity, you’ll need to use code.
Given its age and popularity, Selenium has a large community behind it and offers a large library of add-ons to extend its functionality.
Appium
Based on the specifications of Selenium Webdriver (a core component of Selenium), Appium is best known for automating mobile tests of native, web, and hybrid applications on iOS, Android, and Microsoft Windows devices.
Like Selenium, Appium supports a number of programming languages, including Java, JavaScript, Ruby, Python, PHP, and C#.
Cypress
Cypress is specifically designed for web app testing by front-end developers. Its tests are scripted in JavaScript and run very fast, particularly compared to Selenium tests. It shines in end-to-end testing, but also, technically, can be used for unit testing.
Like Selenium, Cypress includes a test recorder to simplify the creation of basic test steps including actions like click, type, and select. Unlike Selenium, Cypress only supports web testing in Firefox and Chromium browsers (Chrome and Edge) and doesn’t support testing in more than one browser tab or window at a time.
Unlike most of the best-known open-source tools, Cypress offers a commercial product, “Cypress Cloud,” which extends the capabilities of its framework. For example, Cypress Cloud allows you to run your tests in parallel and provides test analytics to give you rich insights into your test runs and failures. (You still have to provision machines to run your tests on.)
Cypress Cloud includes a free plan limited to three users and 500 tests runs per month.
Playwright
Playwright is an open-source, end-to-end testing tool for web apps released by Microsoft in late 2019. Playwright has quickly grown in popularity: as of mid-2022, it has more stars on Github than Cypress.
You can use the Playwright API in many programming languages, including TypeScript, JavaScript, Python, .NET, and Java. It provides cross-browser support for Chrome, (Chromium) Edge, Firefox, and Safari.
Like Selenium and Cypress, Playwright offers a recorder for test creation. It includes screenshots and video recordings of tests without configuring add-ons.
One of the distinguishing features of Playwright is that it automatically waits for elements in your app to be active before taking action. This helps avoid test brittleness you might get in other frameworks that allow hard-coded timeouts.
OpenTest
OpenTest is an open source test automation framework built on top of Selenium and Appium, so it can perform functional tests of web apps, iOS and Android mobile apps, and APIs. It supports running your tests in parallel on local machines or via 3rd-party, cloud-based machines.
OpenTest’s “keyword-driven testing” approach means you can create basic tests with minimal coding skills. For more complex test scenarios, you can code test behavior with JavaScript.
Watir
Watir (Web Application Testing In Ruby) is a wrapper of Selenium Webdriver designed for scripting tests with Ruby. It supports cross-browser testing across all major browsers, but doesn’t currently support native mobile application or API testing. Unlike Selenium, it doesn’t offer a recorder for test creation.
Capybara
Capybara is designed to create simple automated tests for web applications. Its human-readable test scripts are based on the behavior-driven development (BDD) style of frameworks like Cucumber and RSpec.
Capybara supports various drivers, including Selenium and Webkit, with which it can execute tests in a headless browser much faster than Selenium can.
Note: TestProject was listed in a previous version of this post, but all TestProject accounts were deactivated as of March 31, 2023.
Commercial test automation tools with free plans
There are many commercial test automation solutions on the market that resolve at least a couple of the costs that come with open source frameworks.
For example, a number of these commercial solutions include the cloud-based, virtual-machine infrastructure you need to run your tests in parallel at scale. You don’t need to provision any machines to execute tests.
Further, most of these solutions come with record-and-playback functionality, which allows you to create (basic) test scripts without coding skills. You can simply interact with your app and the testing tool will automatically record your actions as test steps, so there’s no need to have expensive technical personnel writing all your tests.
But, generally, these commercial tools still impose several of the same costs that open-source tools do. Every tool is different, though, so it’s important to know the tradeoffs you’re making with each one.
In this section, we’ll examine the handful of commercial tools that come with free plans and explain the costs and limitations of those plans.
Katalon Studio Free
Katalon is a low-code, record-and-playback tool built on top of Selenium, Appium, Cucumber, TestNG, and JUnit libraries. It can test web apps (on all the major browsers), native mobile apps, Windows desktop applications, and APIs. For complex test cases, you can create test steps in the Groovy or Java scripting languages.
Unlike most test automation tools, Katalon provides flexible options for performing visual regression testing (or “UI testing”). You can set checkpoints in your tests to validate a screen, region, or element (like a button or headline) against a baseline screenshot. Then, Katalon can compare your app’s visuals against baseline images using one of three approaches: exact-match, layout-based, or content-based.
You’d expect the free plan of a commercial tool to have limited features, but Katalon Studio Free is more limited than most. On the free version, you can only run tests on a local machine.
You have to pay to get cloud-based, parallel testing across multiple machines, CI/CD integrations (e.g., with Jenkins), test scheduling, and advanced test result insights. If your team and your test suite are very small, these limitations might not be a concern.
Katalon Studio Free’s biggest cost comes with its complexity. Even though you don’t need to know code to create basic tests with Katalon, it is a complex tool that takes time to learn and use. You can’t just jump in and expect to start creating effective tests with Katalon without investing significant time and trial-and-error.
First, there’s the complicated user interface, with its many tabs, nested test elements, and commands. It’s not intuitive, so it’ll take you time with tutorials and practice to figure out what everything means and how it all fits together.
And that doesn’t account for the time you’ll have to spend figuring out how to interpret and edit tests. In Katalon’s “manual view,” there are over 100 different actions that can form the basis for a test step. Some of those actions are intuitive (Click, Delete All Cookies), and some of them are not (Switch to Window Index, Wait for Angular Load.)
How do you know which action to use, depending on what you’re trying to accomplish with your test?
Compare this with Rainforest QA, which was built from the ground up to be intuitive so anyone on your team can start creating and maintaining automated tests right away, without any training.
Rainforest comes with a simple user interface and only 16 test actions to learn. Rainforest doesn’t need as many actions as other, more complicated tools because its actions are more flexible, covering many potential test cases.
These actions are easy to understand and remember because they all correspond with real-world user behaviors (like “Click”, “Scroll”, and “Fill”), not esoteric tool functionality.
Testim
Like Katalon, Testim is a low-code, record-and-playback tool. On its free plan, you can test both web and mobile applications (though you have to upgrade to test APIs). You can customize test scripts with JavaScript.
Unlike Katalon, Testim is relatively straightforward to learn and use. Its visual test editor is largely intuitive and test steps are presented in a timeline view in plain English. Each step includes a screenshot of the element being tested.
Testim’s free plan is more generous than Katalon’s, too. On the free plan, you can run unlimited tests on a local machine, and up to 500 web tests per month on Testim’s cloud of virtual machines. You can also schedule a test and integrate with your CI/CD pipeline on the free plan, but you’ll have to upgrade to run your tests in parallel.
Testim’s biggest limitation is that — similar to open source tools — it performs functional testing, but not visual testing, by default. That means Testim tests evaluate the behind-the-scenes code of your app, but don’t test what your end users will actually see in your app. If your app is technically functional but not usable (e.g., because of a button rendered offscreen or behind another element), Testim wouldn’t alert you.
To add visual regression testing capabilities to Testim, you have to integrate with Applitools, which is only available via Testim’s most expensive plan, starting at $1,000/month.
Rainforest QA, on the other hand, combines functional and visual testing in every test step. Rainforest tests give you the confidence that your app isn’t just functional, but that it’s also providing a good experience to your users.
Unlike most other software testing tools that use locator code in the browser DOM, Rainforest identifies elements in your app (like buttons, links, and form fields) by their visual appearance, just like a real user would. Rainforest then interacts with these elements via the visual layer of your application, just like a real user would.
Rainforest’s approach represents the end user’s experience better than the approach adopted by most other tools, which “sees” code, not the actual visual appearance of your app.
Rainforest’s UI tests use machine learning to compare tested images to baseline images, so minor changes in your app that a human tester wouldn’t notice or care about will be ignored.
This approach also means you can use Rainforest to test anything that appears on the screen of one of our virtual machines — if you can see it, Rainforest can interact with it. So Rainforest tests can download files and validate their contents, change the settings of browser plugins, and more.
Reflect
Reflect is a web-based, record-and-playback test automation tool that offers a free plan. It offers web app and API testing, and allows custom test steps written in JavaScript.
Reflect’s free plan provides up to 30 minutes of test execution time (or ~100 test runs) every month on its cloud-based virtual machines. (You can’t run tests locally or on another test grid.) The free plan also includes CI/CD integration and scheduled tests. You’ll need to upgrade to run your tests in parallel or to run your tests across more than one type of browser (including Chrome, Firefox, Edge, and Safari).
You can add test steps to Reflect that perform visual validations, but they’re not very sophisticated. By default, tested images from your app need to be 99.5% similar to their respective baseline images to pass. You can adjust the matching threshold, but it’s an arbitrary adjustment that can accidentally allow false-positive test passes (i.e., when the test image and baseline image aren’t similar enough that a human tester would consider them a match).
Like most other test automation tools, Reflect uses locators like CSS classes or Xpaths to identify elements in your app. Locators often appear in Reflect’s recorded test steps, making them difficult to interpret.
Reflect is similar to open-source testing software in this respect — having to dig around in the front-end code of your app to figure out which locators belong to which elements can make test maintenance a slow-going drag.
In fact, most teams getting into test automation underestimate how often they’ll have to update their automated tests to keep pace with changes in their app. Anything that makes test maintenance go faster will prevent a lot of pain for your team.
That’s why Rainforest test steps are 100 percent in plain English, so anyone can quickly interpret and update them.
“Free” doesn’t come without costs
As you’ve seen, free tools don’t come without costs.
Open source tools require:
- hiring or allocation of expensive technical personnel,
- provisioning of machines to run your tests on, and
- add-ons to add important functionality like visual regression testing.
Commercial testing software with free plans work around some of these costs, but come with their own:
- Katalon is complex, requiring not-inconsequential training and trial-and-error to use.
- Testim doesn’t do visual regression testing for free, which puts user experience at risk.
- Reflect includes code-based locators in test steps, making them difficult to interpret and maintain.
Rainforest QA comes with none of these costs and challenges. It provides everything you need to allow anyone on your team to start creating and maintaining powerful automated tests for web apps right away without any training.
- It’s no-code and intuitive, including test steps in plain English.
- It does functional and AI-powered visual testing by default, giving you confidence in the user experience of your application.
- It’s a web-based, all-in-one platform, including Windows 10 virtual machines to run your automated end-to-end tests.
- Test results include video recordings, reproduction steps, HTTP logs, and browser logs — everything your dev team needs for debugging.
- Rainforest integrates with email, Slack, and MS Teams for important notifications, and integrates with JIRA for automatic generation of bug tickets.
- It offers test scheduling plus an API, CLI, and CI integrations for continuous testing in your CI/CD pipeline.
Plus, we have affordable plans for individuals and teams who’d otherwise consider the pains and limitations of a free test automation tool.
Talk to us about setting up a Rainforest account so you can try it out.