My name is Asia Wolf, and I’m Imagine Easy’s first QA hire. I’ve been writing automated acceptance tests and performing manual regression tests since April 15th, 2014.
After I studied computer science at Carnegie Mellon University, I got my first job as a Quality Assurance Engineer. This was supposed to be the first step on my way to become a “real” web developer. My task was to automate testing for a e-commerce startup. Knowing nothing about automated acceptance testing, I searched the internet for the right framework to use. Unfortunately, most of the articles I could find were incredibly outdated and only talked about one framework’s good or bad points. Since then, I’ve built 3 automated acceptance testing suites in CasperJS, Selenium with RubyBindings and Appium, and Selenium with JavaBindings as well as doing regular research on the tools on the market such as Cucumber, Behat, and Watir. There are almost endless combinations of frameworks you can use for testing, so I hope that by comparing frameworks and advising based on my experience I can help spare your QA team some frustration in choosing the right framework.
Disclaimer: This is mostly geared towards small automation teams who do not need to support Windows or various versions of Internet Explorer (which should probably be manually tested anyway). For those of you who have to support Windows and IE, this is not going to be terribly informative. You should check out Telerik Test Studio, UFT (detailed below), or Selenium Webdriver 2 with IE bindings.
The good: You can automate most browsers and mobile devices in many different programming languages. It’s pretty widely adopted across other frameworks as well, and can integrate with BDD tools such as Cucumber. There are meetups and conferences and Google Hangouts where you can find support. Selenium is pretty easy to pick up with some programming knowledge. I’ve used Selenium with Java with only minor frustration, and Selenium with Ruby and Rspec combined well with minimal set-up for an easy-to-read output and code. You can also use Selenium Grid/Server or SauceLabs to run your tests in parallel across multiple machines.
The bad: Just because you can do almost anything doesn’t mean you should. It’s very tempting to try to make your test suite run on every browser and support mobile devices, but the maintenance cost is very high and it’ll take you a while to figure out how to make each browser’s driver work with your code. I found that utilizing Selenium’s capabilities to run across browsers and devices and in parallel on Sauce was a lot of effort for very little reward. I can remember only one instance where the other browsers caught a bug, because many cross-browser bugs are visual and won’t be caught by automation. All of that effort you spend maintaining can’t be spent making new tests, and you still have to manually check those browsers for visual bugs.
Selenium Webdriver’s documentation leaves something to be desired. I want easy-to-find API docs with good explanations of all of the things I can do, linked from a Getting Started page; but that is not something I can find with Selenium. There are docs, but you have to dig to get to them.
Selenium 3 was promised in August 2013 for a December 2013 release and has been delayed approximately 6 months so far. An effort to combine it with a W3C spec is causing the lateness. After a quick look at the documentation they’ve released so far, I’m not terribly excited and don’t see myself using it for future automation projects.
You should use this if: There are a few cases for choosing Selenium. Maybe your QA team doesn’t have a lot of programming experience and wants to use the language they’re most comfortable with. It’s easy to get support, and you don’t have to learn a new language’s syntax. This approach is also good if you’re a developer making Selenium tests for your product - you can write in the same language your product is in without having to switch contexts as much. Just don’t overreach and try to make it run in every browser on Sauce Labs.
I would mainly recommend Selenium for large teams who intend to utilize the cross-platform and cross-browser testing features or use Sauce Labs. The maintenance cost will be high, but if you have the patience and bandwidth, go for it. Selenium is a widely-adopted and powerful tool for automation and can help you achieve your wildest automated testing dreams if you’re dedicated enough.
Going headless for your automated tests is, in my opinion, the best choice. Automated tests will most likely not completely replace manual testing, and headless browsing takes less time and minimizes the maintenance burden, because you’re not trying to figure out why your tests don’t work a certain way in one browser. This will leave you time for the inevitable manual testing, or for writing even more CasperJS scripts. For those of you who want to try CasperJS but would rather ease into it from Selenium’s easy-to-see Webdriver, I’d recommend running your scripts on SlimerJS. Whereas PhantomJS uses Webkit, SlimerJS runs on Gecko, and launches a small browser window so you can follow along. SlimerJS is not quite ready for prime-time yet, and through experience I would not recommend using it for large test suites. I look forward to following the progress of SlimerJS in the future.
While CasperJS doesn’t have quite as wide an adoption as Selenium, there are already a lot of great open-source tools to help you do a lot of cool things with it. There has been work to incorporate CasperJS with Mocha, Grunt, and Gulp. We use CasperJS with Travis, our continuous integration tool.
The bad: Again, I’m pretty biased. Because my experience with CasperJS has been so positive, I have a hard time finding bad things to say about it. It’s a bit harder to get answers to your questions on the framework because of its newness, and the version I use is a developer version and thus has some bugs. A lot of the tools that work with CasperJS are very new and still have bugs to work out, so you may invest a good chunk of time into them before realizing that they are not mature enough to use for your testing.
Headless browsing is sometimes intimidating; debugging is not incredibly fun to do when you can’t see what your code is doing in a browser. CasperJS’s debug mode combined with its methods to capture screenshots are a good substitute in my opinion, but be patient when you start learning this framework. You can use SlimerJS to see part of the page you’re on for easier debugging, but the engines run a bit differently and what passes on SlimerJS may not always pass on CasperJS.
The good: Cucumber, Behat, and other BDD tools (including Yadda which plugs into Casper but does not seem quite stable enough for use running tests yet) will restrict the format to combine your acceptance criteria with your automated acceptance tests. It forces you to think through your requirements and makes it easy for non-developers to read your code and run your tests. The syntax is easy to read for people who don’t code.
The bad: Here I have to admit more bias. Every time I start writing a new automation framework I test the part of an app in various languages to compare speed, flexibility, ease of writing tests, and general enjoyment of the language. I’ve never enjoyed writing Cucumber, and find the format kind of limiting. Thus, I have never continued past the experimental stage and actually maintained a cucumber suite.
Many people see poorly developed acceptance criteria and leap to using a BDD tool right away. That might fix the problem, but it will complicate your automated acceptance testing stack unnecessarily. If the root cause is bad acceptance criteria, educate the person writing your acceptance criteria on how you’d like it written, and don’t start writing code until that criteria is clear. Additionally, people see poorly structured automated acceptance tests and leap to BDD. This is again fixing a problem by adding complexity and restrictions, rather than educating your engineers on good practices.
You should use this if: Now, my bias against BDD tools doesn’t completely prevent me from seeing the use. I’ve been a product manager and QA at the same time for the same product, and probably should have used Cucumber instead of Selenium. If your product managers or non-developers read and run your tests, this is probably a good choice. If you are a technical person and everyone reading and running your code is as well, you probably don’t need it. And if you don’t need it, don’t use it and complicate your stack.
Other frameworks and tools I did not include here
I obviously haven’t evaluated every tool on the market. I try not to spout opinions on things I am not informed on, so I can’t pass judgment or advice on the following tools that seem to be decently popular.
Watir: A ruby tool that never really set itself apart enough from Selenium for me to invest the time into investigating it. However, this blog by Allister Scott says some nice things about Watir and makes me wonder if I made the wrong choice, especially given my waning enthusiasm for Selenium.
HP Unified Functional Testing/QuickTest Professional: I run anything but Windows if I have a choice, I like open-source projects, I don’t want to pay for a testing framework, and I have never met anyone who expressed any enthusiasm for this tool. But it does exist and is used pretty frequently at larger companies.
- Look at the disclaimer if you need to automate tests for Windows and Internet Explorer.
There are a lot of tools out there, and what you should use depends on your team size, cross-browser testing needs, and comfort with various programming languages.
For a small team who doesn’t need cross-browser testing, I’d recommend CasperJS.
For a large team with a lot of patience and cross-browser or cross-platform testing needs, I’d recommend Selenium Webdriver 2 with the various browser drivers, Ruby Bindings, and RSpec.
Only use BDD tools if you have a reason.
Only support cross-browser automated testing if you are ready to put the effort into maintaining it.
Want to join our QA team? Email us at firstname.lastname@example.org.