What is the difference between NightwatchJS and WebdriverIO?
This question has become a real fork in the road for a lot of web-automation POC and/or MVP implementations. I don't want it to dictate the wrong direction!
Thus, I feel it needs a more thorough answer, with a broader reach (ease of installation, dependencies, important plugins, services, reporters, documentation, support, & other integrations) and hopefully a more subjective delivery than the accepted answer.
❒ TL;DR: (for the lazies!)
➮ if you're a beginner, starting out with web automation, or you want to build-up the test-coverage for a small/medium-sized web app, then choose any of the two! (might as well do a coin-flip!) The differences between the two are very slim at a lower level. The cracks become gaps only when you reach more complex challenges, later down the road.
➮ if you're looking into building a full-fledged, heavyweight automation harness in order to assure the quality for a big corporate web app, or a massive web portal, then read the entire post (or take my advice & go WDIO!)
❒ Credentials: (what recommends me to have a say in this?)
I have been working with NodeJS based test frameworks extensively since early 2013, having occupied various QA automation engineering roles from entry-level, to senior, to QA Lead. I'll summarize my working experience with these frameworks & web automation:
- WebdriverIO (+4 years)
- Nightwatch (+2 year)
- built, maintained & extended several frameworks using these tools, assuring quality over all major browsers & environments (desktop & mobile)
- kept several WebdriverIO & NightwatchJS talks & presentations at various meetups/conferences
- fairly familiar with the entire NodeJS testing environment (apart from WebdriverIO & Nightwatch, also worked with: WebdriverJS (original fork for WebdriverIO), Nightmare, Robot & as of recently, Puppeteer).
❒ Recommandation: If you have to pick between the two for the implementation of a POC, I recommend you read through this entire answer. Only then can you have a complete outlook on the entire tableau.
❒ Project Details:
.-------------.---------.-----------.----------.-------------.--------------.
| project | ⭐️stars | ððforks | ð«issues| ð¬ open PRs | ð updated |
:-------------+---------+-----------+----------+-------------+--------------:
| webdriverio | 5050 | 1450 | 68 | 6 | 12.Jul.2019 |
:-------------+---------+-----------+----------+-------------+--------------:
| nightwatch | 9450 | 930 | 116|⁺306| 15 | 01.Jul.2019 |
'-------------'---------'-----------'----------'-------------'--------------'
+ - the second value represents the sum of Open Issues ( open label ) & Stale Issues ( stale label, 238 of them ). If you're curious, (I know I was!) stale represents the label given to open tickets that have been inactive for a while, using a bot. (That's pretty suspicious!)
❖ Shallow Takeaways: (this is a personal interpretation of the above numbers, so please take them with a grain of salt)
➮ ( ð¯Strength ) NightwatchJS is a more widely used solution (idea enforced by the total number of Github stars), it also rapidly gained notoriety after being integrated and Evangelized upon by the QA teams in several big companies (e.g: LinkedIn, AirBnB, GoPro, etc.)
➮ ( ð¯Strength ) Due to its robustness, modularity & extensibility, WebdriverIO boasts a higher number of forks. Recent integrations with a lot of popular & on demand services (e.g: ChromeDevTools, React & VisualRegression plugins) have also strengthened the testing community's trust in the solution
➮ ( ðWeakness ) Even though this isn't a tech-specific critique, I lately have become displeased with the constant unprofessionalism surrounding the NightwatchJS team. So, apart from the stale tickets, you can constantly find your perfectly valid BUG, suddenly closed, with little to no information. Sometimes, you get something funny like this.
➮ ( ð±Opportunity ) The next two categories (Open Issues, Open PRs) are actually a true image of the commit-to-contributor ratio: NightwatchJS is mainly maintained by Andrei Rusu and a handful of others, while WebdriverIO stands out as a project led by main contributor Christian Bromann, and backed by a very active & effervescent community.
➮ ( ð£Subjective ) Personally, I sometimes get the feeling that Nightwatch, as a web-automation framework, has been semi-shelved already and the main contributor is more interested in generating revenue from the users, than fixing the current issues. Don't get me wrong... I have nothing against marketing & promoting oneself, but I would much rather put effort back into the product & the respect for my active users, FIRST!
❒ Disclaimer!
The following grades (
on a scale of 1-5 ⭐️) are my personal assessment after having worked extensively with both frameworks. They don't reflect anything other than my own experience with the given tools.
❒ Setting up a Project:
- WebdriverIO (
5/5 ⭐️) - NightwatchJS (
4.0/5 ⭐️)
Setting up a WDIO, or Nightwatch, or any NodeJS-based test framework from nowadays for that matter is pretty straightforward:
➊ install the package:
npm install --save webdriverio
// or
npm install --save nightwatch
➋ install & fire-up a Selenium Server package:
npm install --save-dev selenium-standalone
// don't forget to ...
cd ./node_modules/.bin
./selenium-standalone install
./selenium-standalone start
➌ create a quick test folder structure:
./
├── /test
| ├── /features (test-files go here)
| | └── test.js (your first test here)
| ├── /pages (page-objects go here)
| ├── /assets (everything goes in here)
| └── /custom_commands (custom methods go here)
└── package.json
➍ generating the configuration file (wdio.conf.js, or nightwatch.conf.js):
➮ ( ð¯Strength ) it's at this point that WebdriverIO gets an advantage, as it comes out-of-the-box with a custom-made CLI wizard for an easy & straightforward configuration of the
wdio.conf.jsfile (you can start the wizard via thewdio configcommand)➮ with Nightwatch, you have to copy-paste a mock
nightwatch.conf.jsfile from the Getting Started section➮ it's not that Nightwatch is lacking in this regard, just that I personally feel it leaves more room for interpretation on certain aspects, such as what config keys are mandatory, what is
nightwatch.jsonand how is it different thannightwatch.conf.js, etc. It just seems that after reading the initial config documentation, I leave with more questions than answers.
❖ WebdriverIO's Configuration Helper (CLI wizard):
❒ Writing & Running our first Test:
- WebdriverIO (
5/5 ⭐️) - NightwatchJS (
4.5/5 ⭐️)
OK, you have now finally set up your project, you have all your software dependencies in check, not it's time to write our first test. Your project at this current step should look like this:
./
├── /test
| ├── /features
| | └── test.js
| ├── /pages
| ├── /assets
| └── /custom_commands
├── wdio.conf.js or nightwatch.conf.js
└── package.json
I really like & appreciate that both Nightwatch & WebdriverIO have some nice, out-of-the-box, working examples on their respective homepages. Plug & play!
❖ NightwatchJS example: (tests a Google search for Rembrandt)
module.exports = {
'Demo test Google' : function (client) {
client
.url('http://www.google.com')
.waitForElementVisible('body', 1000)
.assert.title('Google')
.assert.visible('input[type=text]')
.setValue('input[type=text]', 'rembrandt van rijn')
.waitForElementVisible('button[name=btnG]', 1000)
.click('button[name=btnG]')
.pause(1000)
.assert.containsText('ol#rso li:first-child',
'Rembrandt - Wikipedia')
.end();
}
};
❖ WebdriverIO example: (tests a DuckDuckGo search for WebdriverIO)
❯ wdio-v5 example:
describe('DuckDuckGo - Search Test, () => {
it('Should test the DuckDuckGo search page', async () => {
await browser.url('https://duckduckgo.com/');
await browser.setValue('#search_form_input_homepage', 'WebdriverIO');
await $('#search_button_homepage').click();
//
const title = await browser.getTitle();
assert.equal(title, 'WebdriverIO at DuckDuckGo', `Checking the title ...`);
});
});
❯ wdio-v4 example:
const webdriverio = require('webdriverio');
const options = { desiredCapabilities: { browserName: 'chrome' } };
const client = webdriverio.remote(options);
client
.init()
.url('https://duckduckgo.com/')
.setValue('#search_form_input_homepage', 'WebdriverIO')
.click('#search_button_homepage')
.getTitle().then(function(title) {
console.log('Title is: ' + title);
// outputs: "Title is: WebdriverIO (Software) at DuckDuckGo"
})
.end();
This becomes especially handy when you want to show someone a fast test framework deployment, or teach an automation course & you're missing a working example to test the work at hand.
➮ ( ð¯Strength ) Right before writing your first test, WebdriverIO gives you a decision to make. Do you want to write your code synchronously, or asynchronously? This offers immense flexibility, off the bat, letting you choose the way in which you want to write your tests (using the
syncflag).❗Note: This was a
wdio-v4specific feature! The new WebdriverIO implementation (wdio-v5) focuses on a synchronous approach to test writing, whilst leveraging the asynchronous calls under the hood. For the sake of maintaining a faithful representation of the old version, I'll keep the below examples, though they are no longer valid forwdio-v4.
// By default WebdriverIO commands are executed in a synchronous way using
// the wdio-sync package. If you still want to run your tests in an async way
// e.g. using promises you can set the sync option to false.
sync: true,
❖ sync: true example:
describe('DuckDuckGo - Search Test, () => {
it('Should test the DuckDuckGo search page', () => {
browser.url('https://duckduckgo.com/');
browser.setValue('#search_form_input_homepage', 'WebdriverIO');
browser.click('#search_button_homepage');
const title = browser.getTitle();
console.log('Title is: ' + title);
});
});
❖ sync: false example:
describe('DuckDuckGo - Search Test, () => {
it('Should test the DuckDuckGo search page', () => {
return browser
.url('https://duckduckgo.com/')
.setValue('#search_form_input_homepage', 'WebdriverIO')
.click('#search_button_homepage')
.getTitle().then(function(title) {
console.log('Title is: ' + title)
// outputs: "Title is: WebdriverIO (Software) at DuckDuckGo"
});
});
});
❒ Page Objects:
- WebdriverIO (
5/5 ⭐️) - NightwatchJS (
5/5 ⭐️)
Nowadays it's pretty much impossible having a discussion about web automation & not ending up in a heated discussion about page objects, their usefulness, their implementation, or the Page Object Model in general.
Before we nosedive into how page-objects can be implemented in these two NodeJS test frameworks, I feel we have to understand WHY? we're using them.
WHY? (why are we using page-objects?)
There's a saying, don't reinvent the wheel, so I'm not going to. Instead, I'll quote ThoughtWork's Martin Fawler who IMHO said it best:
"When you write tests against a web page, you need to refer to elements within that web page in order to click links and determine what's displayed. However, if you write tests that manipulate the HTML elements directly your tests will be brittle to changes in the UI. A page object wraps an HTML page, or fragment, with an application-specific API, allowing you to manipulate page elements without digging around in the HTML."
WHAT? (what do page-objects provide us with?)
- they create a separation of concern: test vs. page
- they create a high-level abstraction of the app
- they (should) contains all static information about the page
- they offer the ability to redesign the app w/o changing the tests
- they can represent any element (object) on the page, or the whole page itself
HOW? (how do we create such page-objects?)
Enough blabbering! Let's see how we can easily implement some page-objects inside our test framework.
WebdriverIO page-object example:
❖ page.js (this is your page-object base)
export default class Page {
open (path) {
browser.url(path);
// !Note: if you want to make sure your page was completely
// loaded, then maybe add some logic to wait for
// document.readyState = "complete", or "interactive"
// Use a /custom_command for this! Go D.R.Y. principle! RAWWWWR!
}
}
❖ form.page.js (this is a login form page-object example):
import Page from './page'
class FormPage extends Page {
//
// > define your elements here <
//
get username () { return $('#username') }
get password () { return $('#password') }
get submitButton () { return $('#login button[type=submit]') }
get flash () { return $('#flash') }
//
// > define or overwrite page methods here <
//
open () {
super.open('login')
}
submit () {
this.submitButton.click()
}
}
export default new FormPage()
NightwatchJS page-object example:
❖ homepage.js (this is a homepage page-object example):
const homepageCustomCommands = {
checkHomePage: function() {
this.api.perform((done) => {
//
// > do some homepage checks here <
//
});
return this;
}
};
const homepage = {
url() {
return `${this.api.globals.baseUrl}/homepage`;
},
elements: {
'generic': '#generic',
'elements': '#elements',
'gohere': '#gohere',
sections: {
header: {
selector: '#header',
elements: {
'these': '#these',
'are': '#are',
'your': '#your',
'selectors': '#selectors'
},
},
footer: {
selector: '#footer',
elements: {
// > footer selectors go here <
},
},
},
}
commands: [homepageCustomCommands]
};
module.exports = homepage;
❒ Documentation & Support:
- WebdriverIO (
5/5 ⭐️) - NightwatchJS (
3/5 ⭐️)
Both NightwatchJS as well as WebdriverIO have great support in regards to documentation:
❯ WebdriverIO: Getting Started | API DOCs | Gitter Chat
❯ NightwatchJS: Getting Started | API DOCs
➮ ( ð¯Strength ) both projects have very clean & informative documentation to go along with great examples
➮ ( ð±Opportunity ) as a remark, I sometimes found myself searching for solutions to one, or more Nightwatch feature issues, or framework limitations, only to find the solution on some back-alley gist, or backwater blog. I would be nice if such answers & contributions (like the feature PR submissions left open) would be centralized & documented
➮ ( ðWeakness ) I don't know for sure where the bottleneck resides regarding Nightwatch's slow development cycle & lackluster (at best) support for community BUGs, feature-requests, or even submitted PRs. In my eyes, this appears even more as a flaw in contrast to the vibrant development community around WDIO, or the helpful Gitter chat channel
❒ To be continued ...
❒ Reporting
❒ CI/CD System Integration
❒ Architectural Differences
I've written a test suite using each of these tools a few times.
Webdriver.io allows you to write your test cases "from scratch" and have great control over reporting, by say, integrating with slack using slack npm, and other packages. You would need to know or quickly learn node.js. In addition to working very well with desktop browsers, it integrates well with Appium, Android Studio, and Xcode so you can run your automated tests on Android emulators and iOS simulators locally. You will need to install those things and write some code to tell Appium which drivers to use, and select capabilities, etc.
Nightwatch is a fairly extensive solution that uses an iterator to automatically re-try tests up to three times when they fail. Nightwatch has decent support for integration with VM tools like SauceLabs so that you can theoretically run your test cases against 700+ different platform/browser/version combinations without writing code to manage each driver. Nightwatch handles starting and shutting down selenium for you. While this last sounds very impressive, in reality it's quite a lot of work to attain & maintain that level of test coverage. Nightwatch also has fairly built-in separation of concerns, allowing you to define custom commands and require them in your base test case or individual tests. You could modularize some portions of tests and import them so that you don't have to constantly re-write say, the login test for use in multiple cases. Additionally, you could use custom commands to import selectors as key value pairs.
Having used each I would summarize it this way:
webdriver.io: If you're looking for more control, a very custom solution and you don't need an iterator, and you are confident that you know enough to write code for selecting your browser driver, setting capabilities and you want custom control of your reporting.
Nightwatch: if you want to just start writing tests quickly, to know that it will be relatively easy to run them against specific platforms/browsers/versions and will still allow you significant flexibility to extend your tests by writing custom commands.
Another option out there right now is Dalek.js, which has the easy script creation of Nightwatch but without all the bells and whistles.
Prior to running nightwatch, you can configure browsers in the Magellan.json file, and then when you run your test you call the browsers, or a set of browsers (a "profile") as command line arguments, thusly:
For local browsers:
./node_modules/.bin/magellan --serial --browsers=chrome,firefox
Assuming you have setup a saucelabs account and added you username and access key, you could call a profile of browsers like this:
./node_modules/.bin/magellan --serial --profile=myBrowsers
This assumes you have setup a profile called myBrowsers in the Magellan.json file like this:
{
"profiles": {
"myBrowsers": [
{ "browser": "chrome_46_OS_X_10_10_Desktop" },
{ "browser": "firefox_42_Windows_2012_R2_Desktop" },
{ "browser": "safari_8_OS_X_10_10_Desktop" },
{ "browser": "safari_7_OS_X_10_9_Desktop" },
{ "browser": "safari_9_OS_X_10_11_Desktop" },
{ "browser": "IE_10_Windows_2012_Desktop" },
{ "browser": "IE_11_Windows_2012_R2_Desktop" },
{ "browser": "chrome_45_OS_X_10_8_Desktop" },
{ "browser": "chrome_45_OS_X_10_9_Desktop" },
{ "browser": "chrome_45_OS_X_10_10_Desktop" },
{ "browser": "chrome_45_OS_X_10_11_Desktop" },
{ "browser": "chrome_46_OS_X_10_10_Desktop" },
{ "browser": "chrome_45_Windows_10_Desktop" },
{ "browser": "chrome_45_Windows_2003_Desktop" },
{ "browser": "chrome_45_Windows_2008_Desktop" },
{ "browser": "chrome_45_Windows_2012_Desktop" },
{ "browser": "chrome_45_Windows_2012_R2_Desktop" },
{ "browser": "chrome_46_Windows_10_Desktop" },
{ "browser": "chrome_46_Windows_2003_Desktop" },
{ "browser": "chrome_46_Windows_2008_Desktop" },
{ "browser": "chrome_46_Windows_2012_Desktop" },
{ "browser": "chrome_46_Windows_2012_R2_Desktop" },
{ "browser": "firefox_42_OS_X_10_9_Desktop" },
{ "browser": "firefox_42_Windows_2012_R2_Desktop" },
{ "browser": "android_4_4_Linux_Samsung_Galaxy_S4_Emulator", "orientation": "portrait" },
{ "browser": "ipad_8_4_iOS_iPad_Simulator", "orientation": "landscape"},
{ "browser": "ipad_8_4_iOS_iPad_Simulator", "orientation": "landscape"},
{ "browser": "ipad_9_0_iOS_iPad_Simulator", "orientation": "landscape"},
{ "browser": "ipad_9_0_iOS_iPad_Simulator", "orientation": "portrait"},
{ "browser": "ipad_9_1_iOS_iPad_Simulator", "orientation": "landscape"},
{ "browser": "ipad_9_1_iOS_iPad_Simulator", "orientation": "portrait"},
{ "browser": "iphone_9_1_iOS_iPhone_Simulator", "orientation": "portrait"},
{ "browser": "iphone_9_1_iOS_iPhone_Simulator", "orientation": "landscape"}
]
}
}
SOME OF THE MORE USEFUL (optional) COMMAND LINE ARGS:
toggling the --serial argument results in the test execution being serialized and with more verbose test experience where you can review the errors that have been returned during a run. It also take much longer to run as it wait for tests to complete.
adding the --sauce argument once your test cases work for browsers that exist on your local machine, you can tap into the (currently) 760 browsers supported by Sauce Labs. Go ahead an paste this into the terminal & hit return:
./node_modules/.bin/magellan --serial --list_browsers
For each device/browser you want to test you just add the listing in the Copy-Paste Command-Line Option column as comma separated values after --browser= when executing the script. NOTE: when running without --sauce you can just use --browser=chrome or --browser=chrome,firefox
BREAKING IT DOWN:
Using nightwatch without the --sauce but with --serial is a great way to get started. Work on your script until you have validated the things you want to check and when you are confident that all tests that should pass, do, run it with sauce labs and the major browsers you want to test. Once you are confident that the major browsers are covered you can run it without --serial to reduce run time (useful on Sauce Labs, which will cost money).
But enough proselytizing, you can find out what you need about Saucelabs here: https://wiki.saucelabs.com/display/DOCS/The+Sauce+Labs+Cookbook+Home
And for a boilerplate example of Nightwatch to do the canonical hello world: try this boilerplater
UPDATES: A few points that people are bringing up and that have occurred to me since posting this.
Webdriver.io:
Since there is no iterator, there is less ability to recover from failures during a test execution, this means failures are more definitive. Because this is purely async, you may have headaches tracing the exact origin of the failure. You may also end up having to create separate tear-down scripts for any data you create to avoid data collisions during execution.
Nightwatch.js: Since the iterator allows you retries, you will often be able to find where your script is failing. This may allow you to more quickly find the defect instead of focusing on why your script is failing. It's also easier to toggle off your individual scripts.
UPDATE 2:
With Nightwatch shorter tests are useful/encouraged. Because the iterator reads the test files into memory each and every iteration immediately before execution, you can very literally edit the tests between iteration execution. Let me say that a different way: Your Nightwatch suite:
test_1 starts
test_1 FAIL // because you made a trivial error in your test case
test-2 starts // while it is running, you make the change, save it
test-2 PASS
test_1 starts // the iteration starts * with your change! *
test_1 PASS
============= Suite Complete =============
Status: PASSED
Runtime: 2m 48.3s
Total tests: 2
Successful: 2 / 2
1 test(s) have retried: 1 time(s)
On the other hand, setting up Slack webhooks with node/webdriver.io is easy. What this means is that you can set up your node/webdriver.io tests to report to Slack when they complete. Clients appreciate this because after a build has completed they soon see the results of the automation, like:
✅ Automated testing of [client/product name here] Sprint ##.#.# passed on [server URL or IP address] with OS X Firefox 59.0.2
❌ Automated testing of [client/product name here] Sprint ##.#.# failed on [server URL or IP address] with OS X Firefox 59.0.2
UPDATE 3 (August 6, 2017)
Having spent another year and half working with both on a daily basis, I want to add the following points.
There are similar numbers of NPM packages out there that integrate with each, but you'll note that there are far more Stackoverflow questions about Nightwatch (4x). I believe this is because Webdriver.io is more of a roll-your-own approach to automated testing [that's my opinion, and I welcome feedback/pushback]. Those who use it won't have questions about how to use it, they'll have specific questions about techniques.
Nightwatch would be a better entry point for someone with extensive Selenium IDE and solid javascript experience. It has a lot of useful solutions out of the box. What little experience I have with Dalek suggests that would similarly be a good option.
Someone with more javascript and perhaps some Object Oriented Programming and unix experience would likely find Webdriver.io to be better. It's just a great option to build your own custom framework, as I am currently doing. If you can envision how you would like your initialization, flow control and reporting to work, and are willing to put in the sweat equity, it's apt.
I was asked below which I prefer, and I prefer Webdriver.io for extensive e2e tests by far. While I often use a personalized Nightwatch repo for most client work built on top of our platform, that may change in the near future as I build my own Webdriver.io solution.
UPDATE 4 (May 2, 2018)
Updated for clarity about controlling Selenium and browser drivers as well as adding some details about using Appium and Xcode/Android Studio.