2008-12-19 23:03
Two excellent open-source tools for testing web applications are under development: Watir and Selenium. I've been contributing to both. Why? Wouldn't it be better to just have one good tool?
Both tools run tests directly in a browser, and both do it in a way that allows the browser to be minimized while the tests are running, which means you don't have to dedicate a machine to running tests. And both have been used to test significant web applications under development. But there are also many differences'which ultimately stem from the different goals they were originally designed to meet.
Watir was developed as a consequence of the scrīpting for Testers class originally developed by Brian Marick and me. We conceived the class as a way to help get testers over the programming hurdles to do automated testing. Ruby's simple syntax and interactive shell (IRB) made it a simple language to teach. Of the various automation tasks we originally presented, we found that the students were most interested in driving a browser. And the class has come to center on browser-based testing. Watir is but the latest version of the web-browser driving library used for that class. Watir was designed both to be easy to learn, without making any assumptions about the application being tested. For example, it allows page elements to be identified by index, name, ID, value or adjacent text. However, the technical approach we used for the tool restricts it to only working with the IE browser on Windows. As a tool originally intended for instructional use, we were willing to live with this limitation. Well, actually, I was more willing than Brian, who has since become a Mac user.
Jason Huggins, the creator of Selenium had very different objectives. His foremost goal was to create a tool that would allow him to test for browser compatibility. He was developing the time tracking system used inside ThoughtWorks. For in house applications, most IT groups pick a browser, test the app on it, and then tell their users what to use. It's not that easy at ThoughtWorks, however. Our developers are an ornery bunch and don't like people to tell them what tools to use. Although the company provides Windows laptops, many even use Macs or Linux as their preferred platforms. So it was important to Jason that he be able to test multiple browsers on multiple platforms. As other developers at ThoughtWorks have adopted and developed Selenium, they have also been willing to adapt their web applications to the requirements of the tool.
In short, Watir was designed for ease of use'but with the expectation that its users will be programming (rather than recording). Originally intended as a teaching tool, it was acceptable for the tool to be limited to IE and Windows. Selenium was designed for breadth of coverage. It was expected to be used by the same developers who built the application, so it's biggest requirement'that it be installed with the application under test'was acceptable.
Many Watir users report that they are able to quickly get up and running and productive with Watir even if they don't know Ruby. The public reception of Selenium is that it's a cool idea that shows promise.
The biggest complaint with Watir regards it being limited to IE and Windows. The biggest complaint with Selenium is the requirement for server-side installation. The other major difference between the tools is with the novelty of the architecture used. Watir has an architecture that has often been used before; it benefits from the experiences of many people who have built similar tools before. Selenium, on the other hand, represents an original invention. As far as i know, no one has ever built a tool like Selenium before. This brings with it some growing pains as people get experience using its unique approach.
Technically, Watir is a Ruby library that wraps the COM interface to Internet Explorer. COM is a long-standing Windows-based technology for making libraries accessible to various languages. (OLE and ActiveX are other names for approximately the same thing as COM.) Like most languages, Ruby can access COM interfaces. The COM interface to IE allows access to the browser's document object model (DOM). By directly accessing the DOM, we don't have to worry about screen locations or whether the element we wish to access is currently obscured by another window. Using the DOM allows Watir-based tests to run in minimized browsers. Many people have built tools to access the DOM via IE's COM interface; i call these tools COM/DOM drivers. Samie and Jiffie are two other open-source COM/DOM drivers, written in Perl and Java, respectively.
The main thing that distinguishes Watir from other COM/DOM drivers is the way Paul Rogers and i have designed it to meet the expectations of non-technical users. Unlike Samie, it automatically anticipates when it should wait for the next page to load in the browser. It also provides several commands intended to be used with Ruby's interactive shell (called IRB), a language feature found in Ruby'and Python' but not many of the other popular programming languages. Being able to use these commands from the interactive shell helps understand the structure of the web pages they are writing tests for. Watir also has benefited from a growing community, contributing features and documentation.
Selenium uses a unique mechanism for driving browsers. Selenium's automation engine is written in Javascrīpt and runs inside a browser. All modern browsers support Javascrīpt, and therefore can be driven by Selenium. The automation engine, called a browser bot, is embedded in a page that accessed the application under test in a separate frame. There's one snag. Because of something called cross-site scrīpting, Selenium's browser bot has to be served from the same site as the application under test. Thus it has to be installed on the server.
Although Selenium is written in Javascrīpt, Selenium tests are not. Rather, Selenium supports a command language, called Selenese. Tests can take one of two forms. They can be embedded in an HTML table, which will be read in by the browser bot and then executed. These tests look a lot like Fit-based tests, with the Selenese commands amounting to a set of action words.
You can also express Selenium tests in a programming language, taking advantage of language-specific drivers that communicate in Selenese to the browser bot. Java and Ruby drivers have been released, with dot-Net and Python drivers under development. These drivers allow you to write tests in Java, Ruby, Python, C#, or VB.net.
The idea that Selenium has to be installed on the server has turned off a lot of testers. One called it 'scary.' Developers don't seem to mind. One practical concern regards ease of installation. Developers who are used setting up servers don't seem to have much trouble figuring out how install Selenium on their particular server configuration. With time and experience we'll see more documentation on how to install Selenium with typical web application architectures. And we'll see Selenium packaged to be easier to install. But the testers' resistance seems to go beyond the practical concerns with installability. Some testers are reluctant because they don't regularly have direct access to the servers they test. I think others are scared that it will somehow contaminate the validity of the test results. I don't share this concern.
I also won't be surprised if we eventually see a method for installing Selenium off the server. Prototypes have been built for mechanisms that fool the browser into thinking the web app is hosted with the Selenium server even when it's really somewhere else, but these haven't been released yet.
With time, we'll also see Selenium support more browsers, such as Safari and Opera. Although the browser bot can run in these browsers, it needs to be customized for different browsers, because they all seem to expose their DOM slightly differently. The nice thing about Selenium is that this customization is all in the browser bot itself; Selenese commands work with any supported browser.
My plan is to provide multi-browser and platform testing capability to Watir by allowing it use Selenium's Ruby driver to drive browsers in addition to the IE/COM interface that it currently uses. One challenge will be that IE/COM provides a much more fine-grained interface than currently provided by Selenium. I anticipate this project will require new Selenese commands as it progresses. It'll take a while.
In the mean time, users must choose between a tester-friendly tool that is limited to IE on Windows and a tool that works with most browsers but isn't as easy to use or install.