Comparison between Selenium, Protractor, Cypress, and WebdriverIO

The idea of the document is to compare different automation tools that can be used for setting up the automation framework for functional E2E tests. The following automation tools will be compared:

1. Selenium — https://www.selenium.dev/
2. Protractor — https://www.protractortest.org/#/
3. Cypress — https://www.cypress.io/
4. WebdriverIO — https://webdriver.io/

Introduction

First of all, let’s begin with a brief introduction of each of the automation tool that we are exploring:

Selenium

Selenium is an open-source testing tool that allows users to test web applications across different browsers and platforms. Selenium includes a suite of software that developers can use to automate web applications including IDE, RC, WebDriver and Selenium grid, which all serve different purposes.

Since Selenium is open-source, there is no licensing cost involved, which is a major advantage over other testing tools. Other reasons behind Selenium’s ever-growing popularity are:

• Test scripts can be written in any of these programming languages: Java, Python, C#, PHP, Ruby, Perl & .Net
• Tests can be carried out in any of these OS: Windows, Mac or Linux
• Tests can be carried out using any browser: Mozilla Firefox, Internet Explorer, Google Chrome, Safari or Opera
• It can be integrated with tools such as TestNG & JUnit for managing test cases and generating reports
• It can be integrated with Maven, Jenkins & Docker to achieve Continuous Testing

Protractor

Protractor is an end to end testing framework specially designed to perform automation testing on AngularJS based web applications. Protractor API works as a wrapper over the most powerful Selenium WebDriver API and is built on top of WebDriverJS that uses native events and browser-specific drivers to interact with web-based applications as a normal user would do.

There’s a massive increase in locator strategies in Protractor which also allows you to test AngularJS specific elements including the existing locator that we get from WebDriver API without configuring or putting any extra effort. Waits and Syncs are handled in a more effective manner so we need to worry about sync issues as well.

Protractor is one of the ends to end testing framework in AngularJS. Salient features of the Protractor Automation tool:

• Built on the top of WebdriverJS and Selenium server
• Introduced new simple syntax to write tests
• Allows running tests targeting remote addresses
• Can take advantage of the Selenium grid to run multiple browsers at once
• Can use Jasmine or Mocha to write test suites

Cypress

Cypress is a next-generation front end testing tool built for the modern web. We address the key pain points developers and QA engineers face when testing modern applications. Cypress is most often compared to Selenium; however, Cypress is both fundamentally and architecturally different. Cypress is not constrained by the same restrictions as Selenium.

Cypress enables you to write all types of tests:

• End-to-end tests
• Integration tests
• Unit tests

Cypress consists of a free, open-source, locally installed Test Runner and a Dashboard Service for recording your tests.

• First: Cypress helps you set up and start writing tests every day while you build your application locally. TDD at its best!
• Later: After building up a suite of tests and integrating Cypress with your CI Provider, our Dashboard Service can record your test runs. You’ll never have to wonder: Why did this fail?

Cypress comes fully baked, batteries included. Here is a list of things it can do that no other testing framework can:

• Time Travel: Cypress takes snapshots as your tests run. Hover over commands in the Command Log to see exactly what happened at each step.
• Debuggability: Stop guessing why your tests are failing. Debug directly from familiar tools like Developer Tools. Our readable errors and stack traces make debugging lightning fast.
• Automatic Waiting: Never add waits or sleeps to your tests. Cypress automatically waits for commands and assertions before moving on. No more async hell.
• Spies, Stubs, and Clocks: Verify and control the behavior of functions, server responses, or timers. The same functionality you love from unit testing is right at your fingertips.
• Network Traffic Control: Easily control, stub, and test edge cases without involving your server. You can stub network traffic however you like.
• Consistent Results: Our architecture doesn’t use Selenium or WebDriver. Say hello to fast, consistent and reliable tests that are flake-free.
• Screenshots and Videos: View screenshots are taken automatically on failure, or videos of your entire test suite when running from the CLI.

Webdriver IO

WebdriverIO lets you control a browser or a mobile application with just a few lines of code. Your test code will look simple, concise, and easy to read. The integrated test runner lets you write asynchronous commands in a synchronous way so that you don’t need to care about how to handle a Promise to avoid racing conditions. Additionally, it takes away all the cumbersome setup work and manages the Selenium session for you.

Working with elements on a page has never been easier due to its synchronous nature. When fetching or looping over elements you can use just native JavaScript functions. With the $ and $$ functions WebdriverIO provides useful shortcuts that can also be chained in order to move deeper into the DOM tree without using complex XPath selectors.

The test runner also comes with a variety of hooks that allow you to interrupt the test process in order to e.g. take screenshots if an error occurs or modify the test procedure according to a previous test result. This is used by WebdriverIOs services to integrate your tests with 3rd party tools like Appium.

Trade-offs/Drawbacks of each tool

Selenium

1. Setting up the framework — If we use selenium for Python or Java, then it can take considerable time in setting up the framework.
2. Selenium tests can be flaky: A true flaky selenium test will do something like pass on the first run, then fail on the next, then pass again on the third run, all while executing the same exact flow on the same exact environment/region/setup/config/etc. In many ways a flaky Selenium test can actually be worse than no test at all as it gives about the same information to the tester, only now the test engineer must spend hours or days to debug the existing flaky test.
3. Timeout or Sync Issue — Whether you call it a timeout or sync issue, it is one of the most common challenges in Selenium test automation. If you don’t handle this issue carefully, most of your testing script might fail. It is even proved many times that around 80% of scripts fail due to improper sync while executing automation testing. Though this problem can be avoided using smart waits like Implicit waits, Fluent waits, and explicit waits that are available within Selenium.
4. Selenium tests can be hard to debug — If we have not incorporated proper logging, capture screenshots at necessary intervals, selenium tests can be hard to debug
5. Selenium tests can be slow — If proper guidelines are not followed and we add more sleep/waits, selenium tests can get slow over time with new scripts being added it can further slow down the script execution.

Protractor

1. Protractor is not updated — Protractor was not actively updated for a few years. No major improvements, no bug fixes, no documentation updates. You can see tons of old issues in the main repo and absolutely no movement in commits for a few months.
2. Protractor 6 breaks compatibility — In Protractor 5 promises were synchronized using the control-flow mechanism. This was also taken from the selenium-webdriver library. However, selenium-webdriver is dropping support of control flow in version 4. This means that once Protractor is upgraded to selenium-webdriver 4 it will drop it as well. So all your tests should be rewritten using the async/await pattern.
3. Protractor is overcomplicated — The next problem of Protractor is the design which is highly inspired by Java and is very complicated for the JavaScript world. Protractor adds thousands of lines of code on top of a very big selenium-webdriver library and when something goes wrong you don’t have any idea why and where the bug has happened.
4. Protractor not driving a good design — The indirect control of web elements with promises everywhere tends to drive design full of chained methods and loops. This makes tests unstable and not readable to humans.
5. Protractor is not required for Angular — Protractor started as a primary tool for the AngularJS framework. It was the most popular framework for single-page applications when there was no React or VueJS. Angular is still widely popular but the truth is that you don’t need Protractor to test Angular application. The only key feature of Protractor for Angular was synchronization. Protractor waits for Angular to finish rendering before taking any action on-page. This worked fine in the era of AngularJS 1 but it is less and less stable as Angular evolves. Modern Single Page Applications are built around components and there is no single point of truth if a component has finished rendering or not. So instead of relying on Angular to synchronize components by some magic, it is more reliable to use explicit Selenium waits like. browser.wait(EC.visibilityOf($('#abc')), 5000);

Cypress

1. Automation restrictions — Cypress is a specialized tool that does one thing really well: end-to-end testing web applications while they are under development. You should not use Cypress for things it is not designed for such as Indexing the web, Spidering links, Performance testing, Scripting 3rd party sites.
2. Inside the browser — Cypress commands run inside of a browser. This means we can do things nobody else can. There is no object serialization or JSON wire protocols. You have real, native access to everything in your application under test. It is impossible for Cypress to ‘miss’ elements and it always knows the moment your application fires any kind of event. But what this also means is that your test code is being evaluated inside the browser. Test code is not evaluated in Node or any other server-side language. The only language we will ever support is the language of the web: JavaScript. This trade-off means it makes it a little bit harder to communicate with the back end — like your server or database. You will not be able to connect or import those server-side libraries or modules directly. Although you can require node_modules which can be used in the browser. Additionally, you have the ability to use Node to import or talk directly to your back end scripts using our Plugins API or cy.task().
3. Multiple tabs — There will never be support for multiple browser tabs. Because Cypress runs in the browser, it will never have multi-tabs support. We do have access to the browser automation APIs to actually switch tabs, but there is no reason for us to ever expose them. Most of the time this use case is needed when users click a <a> that opens a new tab. Users then want to switch to that tab to verify that the content loaded. But, you shouldn’t need to do this. In fact, we have recipes of showing you how to test this without multiple tabs.
4. Multiple browsers open at the same time — You cannot use Cypress to drive two browsers at the same time. Just like with multiple tabs — Cypress does not support controlling more than 1 open browser at a time. However, it is possible to synchronize Cypress with another back end process — whether it is Selenium or Puppeteer to drive a 2nd open browser. We have actually seen this work together quite nicely! With that said, except in the most unusual and rare circumstances, you can still test most application behavior without opening multiple browsers at the same time.
5. Same-origin — Each test is bound to a single origin. Each test is limited to only visiting domains that are determined to be of the same-origin. What is same-origin? Two URLs have the same origin if the protocol, port (if specified), and host are the same for both. Cypress automatically handles hosts of the same superdomain by injecting document.domain into text/HTML pages, so a host of the same superdomain is considered fine.

Webdriver IO

1. Debugging — Tasks written in this beautiful Selenium API can only be debugged using the provided WDIO task runner. You can’t set breakpoints within tasks, but you can have WDIO pause the run between Selenium commands.
2. Documentation — The documentation is not too great and is incomplete.

The detail below highlights the detailed differences between the 4 tools: