The Good, the Bad, and the Ugly, Reservoir Dogs, Pulp Fiction - these are films you’ve probably watched at least once in your life, or if you’re like me, you watch them once a year.
You might be wondering - I wanted to compare the most widely used E2E automation tools today to help me pick a tool to test web applications with, so why start by listing famous movies? The reason is simple: like the characters in those films, the automation tools mentioned in the title are now in a classic Mexican standoff.
A Mexican standoff is a point where several characters come to a head and point weapons at each other. The characters are usually on equal footing, and it’s rare for everyone to leave the standoff alive. Don’t worry; in this case, everybody will leave the standoff alive with new knowledge and a better understanding of our titular characters.
In the current world of E2E testing, three tools are most frequently mentioned: Cucumber, Playwright, and Selenium. What if I told you that if you combine their capabilities, you can achieve a more effective testing strategy that meets all the needs of modern web application testing?
The goal of this blog is to describe each tool, highlighting their pros and cons, compare them, and demonstrate that they don’t have to compete against each other but can work together through integration.
1. Cucumber - The reliable choice?
Cucumber is a Behaviour-Driven Development (BDD) tool that supports writing human-readable tests using a domain-specific language. It provides a platform that bridges the gap between stakeholders and technical teams.
In BDD, business analysts and/or product owners write Cucumber scenarios that describe the system's behavior from the customer’s perspective. Product owners then review and sign off on these scenarios before developers start writing their code.
The Cucumber platform uses a native Selenium WebDriver method as a framework for automating tests on web applications. It was originally built using the Ruby programming language. However, because of its popularity, it has also been implemented with other multiple programming languages in mind.
Traditional testing approaches are typically code-based, making them more challenging to maintain and update. In contrast, Cucumber tests are designed to be flexible and easier to maintain over time. They are a reliable choice, as they are less likely to break when new code changes are introduced.
The Cucumber testing platform is structured around feature files and step definitions, which utilize specific keywords to give tests structure and meaning. This structure provides clarity and purpose to the steps executed when users write test scripts, which in this case are called Cucumber scenarios.
Cucumber scenarios are written using the Gherkin syntax and stored in .feature files. As you will soon learn in this blog, the test code behind Gherkin is written in the programming language specific to the project. Step definitions are an essential part of the process that connects Gherkin with the code and elements within the application. These definitions are the glue that helps bring various parts of the testing process together. They define the actions that the application will take in response to the user’s input. Overall, step definitions are a critical component of the testing process and play a crucial role in ensuring the quality and reliability of the application.
2. Playwright - The versatile performer?
Playwright is an open-source framework for web testing and automation designed by Microsoft.
Playwright is supported by Linux, macOS, and Windows. It uses a single API, which allows users to write tests using the same syntax regardless of the browser on which scenarios are tested.
Playwright primarily supports JavaScript, TypeScript, and Python. It offers cross-browser testing support, allowing users to run tests on multiple browsers, including Chromium, Firefox, and WebKit-based browsers, without needing to switch frameworks. Playwright also allows parallel test execution to the user, which speeds up the tests and reduces overall testing time.
Using independent browser contexts, each browser instance created by Playwright operates within its own isolated environment. This isolation allows for separate browser sessions with specific cookies, cache, and local storage, enabling efficient and secure scenarios along with parallel test runs.
Additionally, Playwright allows users to intercept and modify network requests made by the browser, mock responses, simulate network errors, or dynamically modify request parameters.
3. Selenium - The industry standard?
Selenium is an open-source set of tools and libraries for browser automation. Like its two opponents, Selenium also supports cross-browser testing.
The Selenium community continually works towards the goal of “One Selenium script for all browsers.” According to StatCounter, the most widely used browsers globally are Chrome, Firefox, Safari, Internet Explorer, Opera, and Edge. Selenium scripts are compatible with all these browsers, meaning you only need to write one script that works smoothly across different browsers, eliminating the need to rewrite scripts for each one.
It provides a single interface that lets you write test scripts in programming languages like Ruby, Java, PHP, NodeJS, Perl, Python, JavaScript and C#. Selenium's scalability ensures comprehensive test coverage, while reusable test scripts save time and effort.
Parallel testing execution is a big factor in the Selenium toolset, which allows the user to run multiple tests simultaneously. This is a crucial factor in speeding up the process and making the development process more efficient.
With detailed reporting and documentation, Selenium also provides detailed execution logs, making it easier to track run results and potential areas that require inspection.
Selenium consists of 4 main components. Let's go over them.
Selenium IDE
Selenium IDE is a Chrome and Firefox plugin designed to record user interactions like clicks and selections in the browser and play them back as automated tests. It generates test scripts in various programming languages, including C#, Java, Python, Ruby, and Selenese (Selenium’s own scripting language).
Selenium RC
Selenium RC was created to automate web application testing by simulating user interactions across different browsers and platforms. It enabled remote browser automation and supported test scripts written in various programming languages.
Selenium WebDriver
Selenium WebDriver is an advanced version of Selenium RC designed to overcome its limitations. It communicates directly with browsers using browser-specific native methods, eliminating the need for Selenium RC.
WebDriver integrates seamlessly with Selenium IDE and Selenium Grid, ensuring reliable and efficient test execution at scale.
Selenium Grid
Selenium Grid is a smart proxy server that enables parallel test execution on multiple machines. It routes commands to remote web browser instances, with one server acting as the hub. This hub directs test commands in JSON format to multiple registered Grid nodes, facilitating efficient and scalable testing.
Playwright vs Selenium vs Cucumber - What is the good and what is the bad side?
What are the advantages of Playwright, Selenium, and Cucumber?
What are the disadvantages of Playwright, Selenium, and Cucumber?
Why have enemies when you can have friends? Integration between automation tools
1. Selenium - Cucumber integration
Many organizations rely on Selenium for their functional testing needs. These organizations are increasingly integrating Selenium with Cucumber to improve the readability and understanding of application workflows.
Cucumber is a tool based on the Behaviour Driven Development (BDD) framework. Its syntax uses Gherkin language, which is written in plain text and serves as a bridge between various roles within a development team.
The simplicity of Gherkin syntax ensures that test scenarios are readable and understandable by all stakeholders, promoting better collaboration and clarity in the development process.
2. Playwright - Cucumber integration
Combining Playwright and Cucumber allows us to use the strengths of both tools to create more powerful and smoother tests. Playwright provides a stable framework for automating browser actions, while Cucumber offers a user-friendly language syntax for describing test scenarios.
Why would you combine Playwright and Cucumber?
Enhanced readability and accessibility - Cucumber's natural, user-friendly language syntax makes test scenarios easy to read and understand
Reliable automation - Playwright provides a powerful framework for browser actions with smooth interaction with web applications
Improved collaboration - Non-technical team members can comprehend and contribute to test scenarios
Early testing adoption - Supports behavior-driven development (BDD) by integrating testing early in the development process
Maintainability - The combined approach promotes easy-to-understand and maintainable tests
What are the best practices for writing test scenarios?
Feature files - Write feature files using Gherkin syntax to describe software behavior
Scenario structure - Use Given, When, and Then keywords to define steps
Step definitions - Implement steps with Playwright’s API to interact with the browser
Data-driven testing - Use tables or regular expressions in Cucumber to run tests with different inputs
What about Playwright vs Selenium integration?
Even though for the past two pairs there are already existing integration, Playwright and Selenium duo are still in experimenting part of their relationship.
Playwright, being a powerful end-to-end testing framework, offers the capability to connect to a Selenium Grid Hub while running Selenium 4. This allows users to launch Google Chrome or Microsoft Edge browsers remotely rather than relying on local instances. However, it is important to note that this feature is currently experimental and should be approached with caution.
How Playwright connects to Selenium Grid
Playwright utilizes the Chrome DevTools Protocol web socket to establish a connection with the browser. Selenium 4 exposes this capability, enabling seamless integration.
However, this may not always be the case since future updates to Selenium could potentially remove this capability, which would disrupt Playwright's ability to connect. Because of this, it is crucial to weigh the potential risks and benefits before adopting this integration in your testing setup.
Risk assessment of the integration of Playwright and Selenium
The integration between Playwright and Selenium Grid Hub carries the risk of breaking in future updates of Selenium. This makes it essential to evaluate the stability and long-term viability of this setup for your projects. Users should take these factors into consideration very carefully to avoid disruptions in testing workflows.
Prepare your Selenium Grid for Playwright
Before you attempt to connect Playwright to your Selenium Grid, ensure your grid functions correctly with Selenium WebDriver. A good starting point is to run one of the example tests provided by Selenium and set the SELENIUM_REMOTE_URL environment variable. If the WebDriver example fails, check the output logs of your Selenium hub, node, or standalone setup for errors and search the Selenium issue tracker for solutions.
Compatibility considerations
Playwright’s connection mechanism relies on the Chrome DevTools Protocol web socket, which is supported by Selenium 4 but not by Selenium 3. While Playwright can still attempt to connect to a Selenium 3 grid, it does so in a best-effort manner by directly accessing the grid nodes.
This requires that the grid nodes are accessible from the machine running Playwright, adding another layer of complexity.
So, should you integrate Selenium and Playwright?
Integrating Playwright with Selenium Grid provides a great way to extend your testing capabilities by using remote browsers. But, given its experimental nature and potential future compatibility issues, it is important to carefully assess the risks and ensure your Selenium Grid is properly configured before proceeding. This preparation will help minimize potential disruptions and ensure a smooth integration process.
So, who is the good, and who is the bad and the ugly in this story?
In the world of E2E testing, Playwright, Cucumber, and Selenium each offer unique strengths. While they may appear to be in competition, like characters in a Mexican standoff from the beginning, they can actually work together harmoniously to enhance your testing strategy.
Cucumber makes test scenarios readable and understandable, fostering collaboration and bridging the gap between technical and non-technical team members. Its BDD approach ensures that all stakeholders are on the same page, promoting clarity and precision in testing.
Playwright offers powerful and reliable browser automation and support for multiple browsers and programming languages. This, coupled with features like isolated browser contexts and network request interception, makes it an excellent tool for comprehensive testing.
Selenium, last but not least, remains a staple in browser automation, known for its cross-browser compatibility and extensive community support. Its components, like Selenium IDE, WebDriver, and Grid, provide a versatile and scalable testing environment.
By leveraging the strengths of each tool, teams can create more readable, maintainable, and powerful tests. This approach not only improves the quality of your testing process but also promotes collaboration and efficiency within your team.
In conclusion, while each tool has its pros and cons, the real power, as always, lies in coming together - their integration. If you have any questions about our testing tools at Devōt, feel free to reach out to us.
