License & Deployment Mix: 16 tools – 12 OSS, 5 SaaS. (OSS and SaaS counts can overlap when an open-source tool also ships a vendor-hosted edition.)
What Is Test Automation Software?
Test automation software covers the frameworks and platforms that run software behaviour as repeatable, machine-checkable tests instead of manual click-throughs. This category spans several layers:
- Behaviour-Driven Development (BDD) frameworks – run plain-language
Gherkin
.featurefiles (Feature/Scenario/Given/When/Then) as executable acceptance tests, mapping each step to a step definition in code (e.g. Cucumber, godog, pytest-bdd, Behat) - Browser / end-to-end (E2E) frameworks – drive a real browser to verify a web app from the user’s point of view (e.g. Selenium, Playwright, Cypress, WebdriverIO, Nightwatch.js)
- Keyword-driven acceptance frameworks – generic, table-driven test syntax with pluggable libraries for web, API, and mobile (Robot Framework)
- Mobile app automation – native / hybrid / mobile-web testing on iOS and Android over the WebDriver protocol (Appium)
- Cloud test grids & studios – commercial platforms that host browsers and real devices at scale, or provide a low-code authoring studio, for the OSS frameworks to run against (BrowserStack, Sauce Labs, LambdaTest, Katalon)
Gherkin is a language, not a tool
A recurring point of confusion: Gherkin is a language – a business-readable syntax for describing behaviour. Cucumber is the tool that reads .feature files and executes them. Gherkin came out of the Cucumber project but is now implemented by a runner per language:
| Language | Runner | Note |
|---|---|---|
| Go | godog | Official Cucumber for Go |
| JS/TS | Cucumber.js | Reference implementation |
| Java | Cucumber-JVM | |
| Python | pytest-bdd, behave | pytest-bdd reuses the pytest ecosystem |
| PHP | Behat | Usually paired with Mink for web |
| Perl | Test::BDD::Cucumber (pherkin) | Feature-complete; on CPAN |
| Raku | Gherkin::Grammar; Cucumis Sextus | Parser + a WIP runner |
.NET is out of scope for this evaluation, so the .NET Gherkin runners (SpecFlow / its successor Reqnroll) are not covered here.
BDD frameworks and browser frameworks are complementary, not competing: a Gherkin runner (Cucumber / godog) supplies the acceptance layer, and a browser driver (Selenium / Playwright) is what a web step definition actually calls.
The information on these pages was researched by a combination of human review and large language models. To suggest an addition or correction, please contact us. Prepared by Rhodium Systems Inc., author of the ResorsIT platform — a unified IT operations management platform for IT teams and MSPs that integrates a curated suite of open-source, commercial, and SaaS applications into a single system with shared identity, single sign-on, access control, and a common audit trail. Use this catalogue only as a starting point for your own research, and review any tool carefully against your own requirements before relying on it. Catalogue data version 2026.197.
Comparison
Tool Categories
Tools are organised by primary role. BDD runners supply the acceptance layer; browser frameworks are what a web step definition drives; the cloud platforms host execution at scale.
| Role | Tools |
|---|---|
| BDD / Gherkin runners | Cucumber, godog, pytest-bdd, Behat, Test::BDD::Cucumber |
| Browser / E2E | Selenium, Playwright, Cypress, WebdriverIO, Nightwatch.js |
| Keyword-driven | Robot Framework |
| Mobile automation | Appium |
| Cloud grids / studios | BrowserStack, Sauce Labs, LambdaTest, Katalon Studio |
Feature Matrix – BDD / Gherkin Runners
| Feature | Cucumber | godog | pytest-bdd | Behat | Test::BDD::Cucumber |
|---|---|---|---|---|---|
Gherkin .feature | Yes | Yes | Yes | Yes | Yes |
| Scenario Outline | Yes | Yes | Yes | Yes | Yes |
| Tags / filtering | Yes | Yes | Yes | Yes | Yes |
| Hooks | Yes | Yes | Via pytest | Yes | Yes |
| Data tables | Yes | Yes | Yes | Yes | Yes |
| Parallel exec | Yes | Yes | Via pytest | Yes | Limited |
| JUnit XML out | Yes | Yes | Yes | Yes | Via harness |
| Cucumber JSON | Yes | Yes | Yes | Yes | Yes |
| Primary language | Multi | Go | Python | PHP | Perl |
Feature Matrix – Browser / E2E Frameworks
| Feature | Selenium | Playwright | Cypress | WebdriverIO | Nightwatch |
|---|---|---|---|---|---|
| Chromium | Yes | Yes | Yes | Yes | Yes |
| Firefox | Yes | Yes | Yes | Yes | Yes |
| WebKit / Safari | Yes | Yes | Exp | Yes | Yes |
| Auto-waiting | No | Yes | Yes | Partial | Partial |
| Network intercept | No | Yes | Yes | Yes | Limited |
| Parallel / shard | Grid | Yes | Cloud | Yes | Yes |
| Cucumber/BDD adapter | Via lib | Via lib | Plugin | Official | Plugin |
| Mobile web | Via grid | Yes | No | Yes | Via grid |
| Runtime | Multi | Multi | Node | Node | Node |
Coverage Matrix – Layer & Target
| Tool | Web UI | API | Mobile | BDD-native | Runtime |
|---|---|---|---|---|---|
| Cucumber | Via driver | Yes | Via driver | Yes | Multi |
| godog | Via driver | Yes | Via driver | Yes | Go |
| pytest-bdd | Via driver | Yes | Via driver | Yes | Python |
| Behat | Via Mink | Yes | No | Yes | PHP |
| Test::BDD::Cucumber | Via driver | Yes | No | Yes | Perl |
| Selenium | Yes | No | Via grid | No | Multi |
| Playwright | Yes | Yes | Yes | No | Multi |
| Cypress | Yes | Yes | No | No | Node |
| WebdriverIO | Yes | No | Yes | Adapter | Node |
| Nightwatch.js | Yes | Yes | Via grid | No | Node |
| Robot Framework | Via lib | Via lib | Via lib | Keyword | Python |
| Appium | Mobile web | No | Yes | No | Node |
SSO / OIDC Comparison
The OSS frameworks have no account system, so SSO applies only to the commercial platforms. The Authentik notes below assume an OIDC/SAML identity provider fronting the tool.
| Tool | OIDC | SAML | SCIM | Authentik Notes |
|---|---|---|---|---|
| Cucumber | N/A | N/A | N/A | CLI framework; tests Authentik-guarded apps |
| godog | N/A | N/A | N/A | CLI framework; no SSO |
| pytest-bdd | N/A | N/A | N/A | Library; no SSO |
| Behat | N/A | N/A | N/A | Library; no SSO |
| Test::BDD::Cucumber | N/A | N/A | N/A | CLI framework; no SSO |
| Selenium | N/A | N/A | N/A | Library; no SSO |
| Playwright | N/A | N/A | N/A | Library; no SSO |
| Cypress | Cloud | Cloud | No | Cypress Cloud SSO on paid tiers |
| WebdriverIO | N/A | N/A | N/A | Library; no SSO |
| Nightwatch.js | N/A | N/A | N/A | Library; no SSO |
| Robot Framework | N/A | N/A | N/A | Library; no SSO |
| Appium | N/A | N/A | N/A | Server; no SSO |
| BrowserStack | Ent | Ent | Ent | SAML/OIDC on enterprise tier |
| Sauce Labs | Ent | Ent | Ent | SAML/OIDC on enterprise tier |
| LambdaTest | Ent | Ent | Ent | SAML/OIDC on enterprise tier |
| Katalon | Ent | Ent | No | TestOps SSO on enterprise tier |
Ent = Enterprise / business tier required N/A = not applicable (local / CLI / library tool)
Deployment Comparison
| Tool | Type | Local | Self-Hosted | SaaS | Docker |
|---|---|---|---|---|---|
| Cucumber | Library | Yes | N/A | No | Yes |
| godog | Library | Yes | N/A | No | Yes |
| pytest-bdd | Library | Yes | N/A | No | Yes |
| Behat | Library | Yes | N/A | No | Yes |
| Test::BDD::Cucumber | CLI | Yes | N/A | No | Yes |
| Selenium | Library/Grid | Yes | Grid | No | Yes |
| Playwright | Library | Yes | N/A | No | Yes |
| Cypress | Library | Yes | N/A | Cloud | Yes |
| WebdriverIO | Library | Yes | N/A | No | Yes |
| Nightwatch.js | Library | Yes | N/A | No | Yes |
| Robot Framework | Library | Yes | N/A | No | Yes |
| Appium | Server | Yes | Yes | No | Yes |
| BrowserStack | SaaS | No | No | Yes | N/A |
| Sauce Labs | SaaS | No | No | Yes | N/A |
| LambdaTest | SaaS | No | No | Yes | N/A |
| Katalon | Desktop+SaaS | Yes | No | TestOps | No |
Grid = Selenium Grid can be self-hosted for distributed runs Cloud = optional Cypress Cloud SaaS for parallelisation / analytics TestOps = Katalon TestOps SaaS for orchestration / reporting
Licensing Comparison
| Tool | License | Type | Notes |
|---|---|---|---|
| Cucumber | MIT | Open source | Multi-language project |
| godog | MIT | Open source | Official Cucumber for Go |
| pytest-bdd | MIT | Open source | pytest plugin |
| Behat | MIT | Open source | PHP |
| Test::BDD::Cucumber | Artistic/GPL | Open source | Same terms as Perl |
| Selenium | Apache-2.0 | Open source | W3C WebDriver baseline |
| Playwright | Apache-2.0 | Open source | Microsoft-maintained |
| Cypress | MIT | Open source | Cloud is proprietary SaaS |
| WebdriverIO | MIT | Open source | Large plugin ecosystem |
| Nightwatch.js | MIT | Open source | Node.js |
| Robot Framework | Apache-2.0 | Open source | Keyword-driven |
| Appium | Apache-2.0 | Open source | Mobile automation |
| BrowserStack | Proprietary | Commercial SaaS | Data egress off-prem |
| Sauce Labs | Proprietary | Commercial SaaS | Data egress off-prem |
| LambdaTest | Proprietary | Commercial SaaS | Data egress off-prem |
| Katalon | Proprietary | Commercial | Free + paid tiers |
Suitability Summary
Recommended for MSP Customers
- Cucumber / godog / pytest-bdd / Behat – match the runner to the customer’s language so acceptance criteria stay readable and executable
- Playwright or Cypress – E2E coverage for customer web apps
- Robot Framework – for teams that prefer keyword-driven, low-code tests
- BrowserStack / Sauce Labs / LambdaTest – only where the customer needs real-device / broad-browser coverage and accepts a SaaS dependency
Caveats
- BDD only checks what someone thought to specify – Gherkin acceptance tests are one gauge, not the whole gauntlet. Pair them with unit tests, mutation testing, and human review for security-sensitive paths.
- Runtime cost – Playwright / Cypress / WebdriverIO / Nightwatch add a Node.js toolchain; Robot Framework / pytest-bdd add Python; Cucumber-JVM adds a JVM. godog is the only runner with no extra runtime for the Go core.
- SaaS grids send test traffic off-premises – a real consideration for a self-host-first, privacy-conscious platform; prefer a self-hosted Selenium Grid where coverage allows.
Tools
16 tools.
Appium
Appium is an open-source cross-platform mobile app automation server driving native, hybrid, and mobile-web apps on iOS and Android via the WebDriver protocol.
License: Apache-2.0 (OSS) · Kind: service · Deploy: package, docker · SSO: none
Behat
Behat is the PHP implementation of Cucumber-style BDD, executing Gherkin .feature files as acceptance tests, commonly paired with Mink for web testing.
License: MIT (OSS) · Kind: library · Deploy: package · SSO: none
BrowserStack
BrowserStack is a commercial cloud testing platform providing on-demand real browsers and mobile devices for running Selenium, Playwright, Cypress, and Appium tests at scale.
License: Proprietary (proprietary) · Kind: service · Deploy: saas · SSO: OIDC, SAML
Cucumber
Cucumber is the open-source Behavior-Driven Development framework that executes plain-language Gherkin (.feature) specs as automated acceptance tests via step definitions.
License: MIT (OSS) · Kind: library · Deploy: package · SSO: none
Cypress
Cypress is an open-source JavaScript end-to-end test framework running in the browser, with an optional Cypress Cloud SaaS for parallelization, recording, and analytics.
License: MIT (OSS) · Kind: hybrid · Deploy: package, saas · SSO: SAML
godog
godog is the official Cucumber BDD framework for Go, running Gherkin .feature files as executable acceptance tests wired to Go step definitions.
License: MIT (OSS) · Kind: library · Deploy: package · SSO: none
Katalon Studio
Katalon Studio is a commercial low-code test automation IDE built on Selenium and Appium for web, API, and mobile testing, paired with the Katalon TestOps SaaS.
License: Proprietary (proprietary) · Kind: desktop · Deploy: package, saas · SSO: OIDC, SAML
LambdaTest
LambdaTest is a commercial cloud test-execution platform offering cross-browser and real-device grids for Selenium, Playwright, Cypress, and Appium test suites.
License: Proprietary (proprietary) · Kind: service · Deploy: saas · SSO: OIDC, SAML
Nightwatch.js
Nightwatch.js is an open-source Node.js end-to-end test framework built on the W3C WebDriver protocol, bundling a test runner, assertions, and page-object support.
License: MIT (OSS) · Kind: library · Deploy: package · SSO: none
Playwright
Playwright is Microsoft’s open-source end-to-end web testing framework with auto-waiting, cross-browser support (Chromium, Firefox, WebKit), and multi-language bindings.
License: Apache-2.0 (OSS) · Kind: library · Deploy: package, docker · SSO: none
pytest-bdd
pytest-bdd is a BDD plugin for pytest that runs Gherkin .feature files as pytest tests, reusing pytest fixtures, plugins, and reporting.
License: MIT (OSS) · Kind: library · Deploy: package · SSO: none
Robot Framework
Robot Framework is an open-source, keyword-driven acceptance test automation framework in Python, with plain-text test data and libraries for web, API, and mobile.
License: Apache-2.0 (OSS) · Kind: library · Deploy: package · SSO: none
Sauce Labs
Sauce Labs is a commercial continuous-testing cloud offering real and virtual browsers and mobile devices for running automated Selenium, Appium, and Playwright tests.
License: Proprietary (proprietary) · Kind: service · Deploy: saas · SSO: OIDC, SAML
Selenium
Selenium is the long-standing open-source browser automation suite (WebDriver, Grid, IDE) for cross-browser end-to-end test automation across many languages.
License: Apache-2.0 (OSS) · Kind: library · Deploy: package, docker · SSO: none
Test::BDD::Cucumber
Test::BDD::Cucumber is a feature-complete Perl implementation of Cucumber-style BDD on CPAN, shipping the pherkin CLI and integrating with Test2 and prove.
License: Artistic-1.0-Perl OR GPL-1.0-or-later (OSS) · Kind: cli · Deploy: package · SSO: none
WebdriverIO
WebdriverIO is an open-source Node.js browser and mobile automation framework built on the WebDriver protocol, with a large plugin ecosystem and a Cucumber BDD adapter.
License: MIT (OSS) · Kind: library · Deploy: package · SSO: none
