Cucumber with using Rspec should_receive

I recently came across a case where I wanted to use a should_receive test in cucumber. The basic idea is that I was attempting to prevent Apache library code from calling out to the system, while simultaneously testing that the correct function was actually called. In Rspec (which backs cucumber in my setup) it is simply:

<script src="https://gist.github.com/1151447.js?file=test.rb"></script>

At the end of the test if restart has not been called then the entire test fails. Because of the way that cucumber works with rspec 1.3 (rails 2.x, which is required by the project) registering a should_receive has no effect (seems this is fixed in the cucumber/rspec versions for rails 3). Even if it were to work it would mean putting a Then before the When or putting a test in a Given. To me that just seems wrong.

<script src="https://gist.github.com/1151447.js?file=apache.feature"></script>

What I had to do to simulate a should_receive but still maintain the flow of the scenario was break it up so that I stub the function and when that function is called I set a flag. Later I test that the flag is set.

<script src="https://gist.github.com/1151447.js?file=apache-after.feature"></script>

The step that sets the flag is Apache can restart and the one that tests is Apache should restart. Here are the step definitions:

<script src="https://gist.github.com/1151447.js?file=apache_steps.rb"></script>

JS Unit Testing using YUI

Ok, Selenium was a partial success that you can read about here, but it really was harder then I wanted it to be and it required a lot of setup for only a little bit of testing.  I noticed that a project that I use a lot ExtJS uses a modified version of YUI Test.  So I converted my Selenium test example to YUI Test.

My initial impression was confusion because you have to download the entire YUI 2 suite just get to get the test tools.  Also, when you download the entire suite they are many different copies of the same file in various stages of minification.  But following the documentation I boiled down exactly what I needed and threw away the rest.  I put all the code for my test here so you can follow along.

You will need the following files from the YUI archive:

  1. build/logger/assets/logger.css
  2. build/logger/logger.js
  3. build/yuiloader-dom-event/yuiloader-dom-event.js
  4. build/yuitest/assets/testlogger.css
  5. build/yuitest/yuitest.js
Create an HTML file that includes the css and js files:
<script src="https://gist.github.com/1151464.js?file=index-header.html"></script>

The HTML will also need to instantiate the test logger and run the test runner global.  I bound the test run to a button so I could control when it ran:
<script src="https://gist.github.com/1151464.js?file=index-body.html"></script>

Now that we have a YAHOO.tool.TestRunner we need to add a test that can be run. Instanciate a new YAHOO.tool.TestCase and add it to the TestRunner.  All a TestCase needs is a name and a bunch of functions that start with "test".  Everything else is automatic.  Below is a simplified version of the full file.

<script src="https://gist.github.com/1151464.js?file=test.js"></script>

Conclusion
Though Selenium is more automated that comes at the cost of being more complicated then testing should be.  If I were a team of programmers then sure setting up and maintaining Selenium Remote controls would be a small part of the overall effort, but since I am not a team of programmers I think it is overkill.

What I really need is an easy to run, easy to write, repeatable, unit testing framework in JS.  I do the leg work on pointing my various browsers at the html and reviewing the results.  When things get big enough that I need to setup a continuous integration server, or I have a QA department, then I will give Selenium another go.  For now YUI test is the way to go.

Learning Selenium

Background
My basic need is to find a platform where I can test FF, IE, and Safari on Windows, Linux, and OS X. I use OS X as my platform, and Safari or Webkit as my environment. I don't like Windows or IE. Linux is OK, but I like OS X because it just works the way I want. And I find FF to be slow, and Firebug which is needed to debug we pages causes rendering changes and timing issues (most notably causing FF to crash).

Ideally I want the testing environment:

  1. To be developed in Safari on OS X 
  2. To be able to test both internal libraries and rendered UI 
  3. To use unit tests to test internal libraries 
  4. To use interactive tests to test rendered UI 
  5. To use the unit/interactive tests as regression system moving forward 
  6. To write the tests once on my browser of choice 
  7. To run the tests on all combinations of browser and platform. 
  8. To not be bogged down by the testing framework 
  9. To be free, or very cheap 
From my research it looked like Selenium did basically exactly what I needed. And unfortunately was the only real option. There were plenty of options for taking screen shots of public sites (which mine isn't, yet) and comparing those between browsers. And there are several options for unit testing javascript, but only Selenium did both and could be run on my own hardware.

Implementation

From my reading it looked like I wanted to use the IDE to create the tests, and remote controls to run the browsers. Eventually I need to scale to Selenium Grid, but that is for later discussion.

Test code
This is the sample file that I created to test.

<script src="https://gist.github.com/1151478.js?file=code.html"></script>

Using the IDE

It took a long time to understand this tool, since I had no background in Selenium. The basic premiss is that each file is a single test case, which is a set of tests. Each test is a grouping of three items: the action, the target, and the expected result. It natively creates a 3 column HTML table, which it can also run, but personal preference is that use the IDE to export the basic test into a different language.

I am a rails developer, and am familar with rspec so I use the IDE to run the tests to make sure they work, but then I transfer it to rspec since it is a more expressive test framework. The downside is that you have to use a Remote Control to run the test, which adds an extra level of complications. We will get to the Remote Control later.

A basic HTML test looks like this:
<script src="https://gist.github.com/1151478.js?file=test.html"></script>

There really is nothing more to it then that. The one thing to note is that the test uses verifyEval which takes a JavaScript string.

In all tests the this object is the base Selenium object so if you want to get to the window object you have to traverse up the stack via this.browserbot.getUserWindow().

Unfortunately everything tested in Selenium is converted to a string before testing. So if I need to ensure that an integer parsing function actually produces a number I need to use typeof.

Using RSpec

The IDE is a great way to test scripts live, but for any programmer it is going to be easier to use a testing framework doing it programmatically. As a rails guy I prefer RSpec so that is what I use.

Installing

This package requires ruby-gems, rspec, and selenium-client. I am going to assume you have ruby-gems installed already. The others are installed like this:
<script src="https://gist.github.com/1151478.js?file=install.sh"></script>

Converting

When using Rspec the only real thing to remember is that there are no assert* or verify* methods. The reason is that Rspec itself is a testing framework so it will have it own version of assert and verify (in this case should).

The IDE has a great feature in that it converts the HTML test into a rspec test for you. It isn't a great format, but it is better then nothing and is a good place to start.

<script src="https://gist.github.com/1151478.js?file=selenium.rb"></script>

The browser being shown is *chrome. This actually means Firefox, not Google Chrome. For Google Chrome use *googlechrome.

For Safari use *safari, but remember that you will need to disable the popup blocker manually and close Safari (for this user) otherwise it will just sit there forever.

Remote Controls

A remote control is what Selenium uses to execute the test. The IDE comes with it built-in, but it is tied to FireFox. To use IE, Safari, or Chrome you need to download the remote control software: http://seleniumhq.org/projects/remote-control. This software is just a Java server that opens your machine on port 4444 (by default) to allow Selenium clients to run tests. Each client gets its own browser instance to run the tests in.

The server must be run by a user that has access the browser and has a screen to render to.

Firefox will only run a single profile per user. If you need to run Firefox concurrently on the same machine you need to fake a second profile. Don't do it, just create a VM; you will be happier.

Google Chrome does not works on OS X. This is because OS X doesn't add application executables to the path, and the server code isn't smart enough to use the executable directly. The fix is supposedly here, but I was not able to get it to work. If I do I will probably write another blog entry and link it here.

Putting it all together

By default RSpec provides no runner code and the code the IDE produces is not standalone. This is not a problem since installing RSpec into a rails app installs script/spec. I have copied the runner code here so make it easier.

<script src="https://gist.github.com/1151478.js?file=runner.rb"></script>

I am going to assume the RSpec runner code is called spec and the test file is called test.rb. To run this test from the command line do the following:
ruby spec test.rb

Assuming you followed all the steps the test should have opened Firefox, executed a page, run the tests, closed Firefox, and returned the results. Now you can add more tests and have Selenium execute them.

Related Research

  1. Using Selenium Grid
  2. Using Chrome, or IE
  3. Using a Grid to run the same test in all browsers on all OSs