A (still brief) experience on using Selenium to test a Rails + ajax app

This is a note to make a point on our (mine and my team’s) current use of Selenium to test the ajax behaviour in the Rails webapp we’re currently developing. Ajax replacing of part of the page is growing, and with it we have to face the classical question: “how do we test (I mean automatically :-) the ajax/javascript behaviours in our webapp?”.

This is how we are trying to manage this issue now, after some days of spiking on Selenium, Watir and BlueRidge (I hope to write more on Watir and BlueRidge in some future post, because these two tools are worth speaking).

Actually we are giving a try to the combination of Webrat + Selenium, since we already have a big test suite of integration test using Webrat, and have a good knowledge of the Webrat API.

We added the selenium-client gem to be able to drive Selenium through the Webrat API.
This is extracted from our test environment configuration file:

test.rb
...
config.gem 'selenium-client', :lib => 'selenium/client'
config.gem "webrat", :version => '>= 0.6.0'
...

Then, we defined a class from which all the selenium test cases will inherit.
This class basically is used to

  • disable the transactional fixtures in Rails, to allow the browser process where Selenium runs to access the data prepared in the tests
  • configure Webrat with the “selenium” mode
  • be the place to collect helper methods as “login” or “logout”, used in many tests.
selenium_integration_test.rb
class SeleniumIntegrationTest < ActionController::IntegrationTest
  self.use_transactional_fixtures = false

  setup :switch_webrat_to_selenium
  def switch_webrat_to_selenium
    Webrat.configure do |config|
      config.mode = :selenium
      config.application_environment = :test
    end

    selenium.set_speed(100)       # default is 0 ms
    selenium.set_timeout(10000)   # default is 30000 ms
  end

  teardown :delete_cookies
  def delete_cookies
    selenium.delete_all_visible_cookies
  end

protected
 ...
 [other helper methods here, like login, logout, and so on...]

 ...

We also added a rake task to be able to launch all the selenium tests

test.rake
namespace :test do
  ...
  ...

  desc "Run Selenium Test"
  Rake::TestTask.new(:selenium) do |t|
    t.libs << "test"
    t.test_files = FileList['test/selenium/*test.rb']
    t.verbose = true
  end
end

One thing we learned through several repeated mistakes is that the Webrat API is different when called in the “selenium” mode then the one we were used to when using Webrat in the classical “rails” mode.
For example, the “assert_have_selector” method for selenium only takes one argument, that is the CSS selector, while in the classical webrat mode, the same method takes another parameter to specify the expected content to match with (see this rdoc: http://gitrdoc.com/brynary/webrat/tree/master). So we had to define helper methods based on “assert_have_xpath” method using xpath to express the same intent of a method like assert_have_selector(css_selector, expected_content)

Here is our helper method

selenium_integration_test.rb
  ...
  def assert_has_id id, text_content
    assert_have_xpath "//*[@id='#{id}'][1][text()='#{text_content}']"
  end
  ...

Fixing SeleniumRC to work with Firefox 3.6

The brand new release of Firefox 3.6 brings, together with some improvements in the browser, also some headaches for all selenium users: actually the latest selenium RC jar (selenium-server.jar) won’t work with Firefox 3.6.

The problem is related to the addons that Selenium will pretend to have in the Firefox instance fired up when Selenium RC server starts. As a matter of fact, those two addons are not declared to be compatible with 3.6.

The simple fix is then to edit the addons’ install.rdf files in the selenium-server.jar to manually set the compatibility to 3.6.

Alternatively, you can download this patched jar from this repository, rename it to selenium-server.jar and replace the previous jar with this.

The actual steps to fix my webrat gem (I use Selenium through Webrat) were

  1. download the above mentioned file (http://github.com/saucelabs/saucelenium/blob/master/selenium-sauce.jar)
  2. rename it to selenium-server.jar
  3. replace the previous file in the vendor folder of your webrat gem (mine was /usr/local/lib/ruby/gems/1.8/gems/webrat-0.7.0/vendor/selenium-server.jar)

One (and a half) useful thing to know when using DeepTest gem with MySQL

DeepTest currently won’t work if you’ve configured MySQL with no password (in other words, if you are able to connect to mysql with a simple “mysql -u root”).
To fix this, you have to patch DeepTest (I know, asap I’ll go through the whole process to propose the patch to the original project leader).
Actually, you have to comment out a line, in the DeepTest:Database:MysqlSetupListener#grant_privileges method:

...
def grant_privileges(connection)
sql = %{grant all on #{worker_database}.*
to %s@'localhost';} % [
connection.quote(worker_database_config[:username])# ,
# connection.quote(worker_database_config[:password])  <-- mysql with no password won't work
]
connection.execute sql
end
...

Another tip (the “half” in the blog post title):
Don’t forget to edit the “pattern” option in your DeepTest rake task, to be able to grab all the testcases you want.
In my case, I want to skip a whole folder containing selenium tests, so I have to write my Deep Test rake file this way:
(in /lib/tasks/test.rake)

require "deep_test/rake_tasks"
...

DeepTest::TestTask.new "deep" do |t|
t.number_of_workers = 2
t.pattern = "test/{unit,functional,integration}/**/*_test.rb"
t.libs << "test"
t.worker_listener = "DeepTest::Database::MysqlSetupListener"
end

ThinkCode.TV goes live!

An advertisement for my friend and mentor Piergiuliano Bossi: his long-waited ThinkCode.TV goes live!

ThinkCode.TV is a website specializes in the delivery of high quality commercial screencasts about software development, at a really cheap prices.

The first 5 videos are now online (the actual language is italian, but english content is planned for the following months):

  • The first two lessions on Python, by Marco Beri
  • A screencast on MacRuby, by Renzo Borgatti
  • The first two lessions on TDD, by Piergiuliano Bossi.

Selling all these high-quality screencasts at about 5 euro each makes them *really* appetible.

Think different about mock objects!

Recently, after the post on mock objects by Uncle Bob (“Manual Mocking: Resisting the Invasion of Dots and Parentheses”), a rather long discussion thread grown in the extreme programming italian newsgroup (starting here, but careful, it’s in italian, sorry!).
This led me to think more deeply about my experience with mock objects, and I’d like to share my point of view here, as it’s quite different (or so it seems to me) from the common opinions on this important topic.

I’ve always followed the so-called (as Giuliano would say, isn’t it Giuliano? :-) “English School” of mock objects, the one coming from the pioneering works of Tim Mackinnon, Steve Freeman and Nat Pryce, the real fathers of mock objects.

And I’ve always carefully followed their advice, first through their *epic* paper  “Mock Roles, not Objects” (http://www.jmock.org/oopsla2004.pdf) – IMHO the best paper on mock objects and on object oriented programming with mocks – then through their terrific posts on the blog www.mockobjects.com, and finally, through their first (and brand new) book, “Growing Object-Oriented Software, Guided by Tests”.

One thing I learn is that mock objects are a design tool, while many people see it only as a technique for speeding up unit tests.
And in this context mock objects are a key tool to support your TDD process, especially in test-driving your domain model, where you follow a process similar to traditional top-down development, in which you start from the highest level of abstraction and then proceed, layer by layer, to reach the core of the domain and then move again towards the boundary of the system, towards the “services” (you can find many similarities in Cockburn’s approach to Hexagonal Architecture).

Then, when you reach the domain boundary, you should stop using mocks.
Mock objects are useful to TDDing the thin adapter layers to your services (that is, third-party libraries or external components (e.g. a database, a JMS queue, …). But then, the actual adapters will be tested with integration testing.

Why?

Because you should use mock objects as far as you can apply TDD, whereas you can design and *discover* interfaces (and roles), and assign responsibility. On the other hand, in front of a third-party library you cannot follow this process, since the code is not under your control, and you cannot modify it.

Because if you use mock objects with third-party libraries (two concrete examples taken from our recent projects: isolating our tests from the database in a Rails app, or in a java app using Hibernate ORM), you’ll write tests that *guess* the library behaviour, and your guesses may be far away from the actual behaviour.
Raise your hands if you never burnt your fingers with this kind of test with mocks, where maybe you *thought* you had a save() method to return an A object while in fact it returned B object! :)

And finally, because this kind of tests with mocks end up to be long, unreadable and fragile (an “invasion of dots and parentheses” reported by Uncle Bob in his post), full of mocks and mock expectations. And, hey, you cannot refactor them, since you don’t own the third-party code!



To verify the correct integration with libraries or external components, which are out of you domain, as well as with integration tests, you may use fakes or stubs (and, by the way, the example in the Uncle Bob’s post is actually a stub, not a “hand-rolled mock”).

So, I’ll repeat myself, following this “mocks as a design tool” approach, you’ll mock only types you own.

Some useful references to study this topic in depth (you’ll be OK even if you read just the first 2-3 links :-)

I hope I give you some useful feedback on this topic!

And, by the way, feedbacks are warmly welcome!

Ruby: how to spot slow tests in your test suite

This is actually my first post in english and also my first post on Ruby/Rails stuff. Twice as hard!

Anyway, we’re working on a Rails project, and we’re experiencing the classical debate in all Rails project (at least the ones with tests!): why our test suite is so damn slow?!
Ok, we know that ActiveRecord is one of the key components in Rails and is at the root of its philosophy of web development. And along with ActiveRecord comes the strong tight between the model and the database. So each test, even the unit tests, will touch the database (ok, technically speaking they cannot be defined unit-tests, I know. Sorry Michael Feathers for betraying your definition).
The very first consequence of this approach is that as your test suite grows with your project, it will become slower and slower.

Let’s take our current project. This is our actual test suite composition:

  • Unit: 317 tests, 803 assertions
  • Functional: 245 tests, 686 assertions
  • Integration: 50 tests, 218 assertions

So we have 612 test methods, for a resulting number of 1707 assertions.
As a side note, our code-to-test ratio is 1:2.3, that is, for each line of production code we have 2.3 lines of tests.
The suite takes about 115 seconds to execute (on my MacBook Pro Core 2 Duo).

So, what can we do to speed up our tests and have a more “feedback-friendly” test suite?
The first step toward the solution of this issue is to have some metrics to reflect on, and so I developed this little ruby module to collect test duration times.
This is how you can use it too:

First, create a file called “test_time_tracking.rb” in the test folder of your Rails project. This should be its content:

module TestTimeTracking
    class ActiveSupport::TestCase
      def self.should_track_timing?
        not(ENV["tracking"].nil?)
      end

      setup :mark_test_start_time if should_track_timing?
      teardown :record_test_duration if should_track_timing?

      def mark_test_start_time
        @start_time = Time.now
      end

      def record_test_duration
        File.open("/tmp/test_metrics.csv", "a") do |file|
          file.puts "#{name().gsub(/,/, '_')},#{Time.now - @start_time}"
        end
      end

    end
end

Then, edit your “test_helper.rb” (again, under the test folder), to require and include the previous module.
E.g.

*test_helper.rb*

ENV["RAILS_ENV"] = "test"
  require File.expand_path(File.dirname(__FILE__) + "/../config/environment")
  require "test_time_tracking"

  class ActiveSupport::TestCase
    include TestTimeTracking
    ...

then, all you have to do is executing your rake task with the “tracking” option set, e.g.
tracking=on rake

At the end of the test suite execution you’ll find a CSV file (test_metrics.csv) in your /tmp folder.
This file contains a line for each test method executed, along with its duration in seconds.
I use to upload this file in google docs, and then apply a formula to sort out the methods from the slowest to the fastest.
A good formula is the following:
=Sort(A2:B612, B2:B612, FALSE)

The main limitation in the current implementation of this module is that every time the suite is executed with rake, the new time metrics collected are appended at the end of the previous file (if it exists), so each time you should remember to move the file to a different location. I’m working on this issue, so I’m expecting to find a better solution. Stay tuned!

Come fare integration test su un plugin per Jira 3.13.4

Quando si tratta di voler scrivere test di integrazione per il vostro meraviglioso plugin per Jira 3.13.4 (l’ultima versione di Jira, in attesa che la 4.0 esca dalla beta), ci si imbatte in una serie di problemi.
Dopo qualche indagine, sono riuscito a risolverli tutti, e mi accingo a condividere la soluzione adottata, nella speranza che possa servire a qualcun altro (anche a me stesso tra qualche mese…).

Premetto che stiamo sviluppato il plugin usando maven2, che per questo genere di cose è davvero molto comodo.
Se avete qualche dubbio, ecco due riferimenti:

Il pom.xml generato da maven usando l’archetipo per plugin di Jira contiene una sezione “properties” che, all’inizio, si presenta così:
<properties>
<atlassian.plugin.key>
com.sourcesense.jira.plugin.MyWonderfulPluginToSaveTheWorld
</atlassian.plugin.key>

<!– JIRA version –>

<atlassian.product.version>3.13</atlassian.product.version>

<!– JIRA functional test library version –>

<atlassian.product.test-lib.version>3.13</atlassian.product.test-lib.version>

<!– JIRA data version –>

<atlassian.product.data.version>3.13</atlassian.product.data.version>

</properties>

Qui trovate una descrizione di queste properties, assieme ai loro valori di default.

Ecco un estratto:

“atlassian.product.version” – version of the Atlassian product to compile and test against.

“atlassian.product.data.version” – version of the test resource bundle that contains the basic Atlassian product configuration data for the integration test environment. These versions mimic the actual Atlassian product versions. However we might only modify and release the relevant projects for the reasons of non-backwards compatibility of the new versions of Atlassian products. Therefore not every version of Atlassian products will have a corresponding version of the resource bundle.

La property “atlassian.product.test-lib.version” non è documentata, e per capire il suo significato dovete chiedere a Google, che vi rispondera’ con questa utile pagina.

“atlassian.product.test-lib.version” – The version of the testing library to use, as a general recommendation you should at least use version 2.0 or higher as it exposes more of the page’s content and provides quite a few extra helper classes to aid in your testing.

Benissimo, quindi io che sto facendo un plugin per la versione 3.13.4 di Jira, sostituisco questo valore nelle tre properties del POM

<properties>
 ...
    <atlassian.product.version>3.13.4</atlassian.product.version>
    <atlassian.product.test-lib.version>3.13.4</atlassian.product.test-lib.version>
    <atlassian.product.data.version>3.13.4</atlassian.product.data.version>
 </properties>

Detto, fatto.
Mi manca solo di creare il mio primo test di integrazione, rigorosamente nel package che inizia con “it”.

 package it.com.sourcesense.jira.plugin;

 import com.atlassian.jira.webtests.JIRAWebTest;

 public class JiraTest extends JIRAWebTest {

    public JiraTest(String name) {
      super(name);
    }

    public void setUp() {
      super.setUp();
      restoreDataWithLicense("JiraDataForTest.xml", ENTERPRISE_KEY);
   }

   public void testVerySimple() throws Exception {
      assertTextPresent("This JIRA site is for demonstration purposes only");
   }
 }

E copiare il dump esportato da Jira per avere qualche dato di test (JiraDataForTest.xml) nel folder src/test/xml/ del progetto del plugin.

A questo punto non mi resta che lanciare il seguente comando nella home della progetto

mvn integration-test

e aspettare con pazienza che maven scarichi quel Terabyte di jar di cui dichiara di aver bisogno.

Primo problema: la console di mvn mi dice

 [INFO] [jar:jar]
 [INFO] Building jar:
        /private/tmp/HelloWorldPlugin/target/MyWonderfulPluginToSaveTheWorld-1.0-SNAPSHOT.jar
 [INFO] [antrun:run {execution: generate-integration-test-config}]
 [INFO] Executing tasks
 [touch] Creating
  /private/tmp/MyWonderfulPluginToSaveTheWorld/target/test-classes/localtest.properties
 [propertyfile] Updating property file:
  /private/tmp/MyWonderfulPluginToSaveTheWorld/target/test-classes/localtest.properties
 [INFO] Executed tasks
 [INFO] [antrun:run {execution: pre-integration-test-user-ant-tasks}]
 [INFO] Executing tasks
 [INFO] Executed tasks
 [INFO] [atlassian-test-harness:start-fisheye {execution: start-fisheye}]
 [INFO] Skipping fisheye; startService is set to false
 [INFO] [atlassian-test-harness:start-confluence {execution: start-confluence}]
 [INFO] Skipping confluence; startService is set to false
 [INFO] [atlassian-test-harness:start-jira {execution: start-jira}]
 [INFO] Output log is set to /private/tmp/MyWonderfulPluginToSaveTheWorld/target/jira/output.log

E si blocca lì.
Vado a vedere il log segnalato nell’ultima riga della console, e scopro una pletora di eccezioni:

2009-07-07 16:11:19,568 main ERROR
[com.atlassian.license.LicenseManager] Exception getting license: java.lang.RuntimeException: contactLicense was null
 at org.picocontainer.defaults.DecoratingComponentAdapter.getComponentInstance(DecoratingComponentAdapter.java:42)
 at org.picocontainer.defaults.SynchronizedComponentAdapter.getComponentInstance(SynchronizedComponentAdapter.java:35)
 ...

Indago, guardo su Google, niente.
Provo allora a sostituire 3.13.4 con 3.13.2 nelle tre properties del POM

<properties>
    ...
   <atlassian.product.version>3.13.2</atlassian.product.version>
   <atlassian.product.test-lib.version>3.13.2</atlassian.product.test-lib.version>
   <atlassian.product.data.version>3.13.2</atlassian.product.data.version>
</properties>

E rilancio “mvn integration-test”.
Stavolta l’errore è più chiaro: fallisce il ripristino del dump JiraDataForTest.xml nell’istanza di Jira 3.13.2 che viene avviata da maven, perchè la versione del dump è stata fatta con la 3.13.4, una versione successiva alla 3.13.2, e quindi Jira si rifiuta da caricarla. Eccheccavolo.

Vi risparmio tutte le combinazioni di numeri di versione che ho provato a mettere nel POM, senza successo, e vado dritto verso la soluzione.
Ecco il pom.xml che funziona

<properties>
 ...
 <atlassian.product.version>3.13.2</atlassian.product.version>
 <atlassian.product.test-lib.version>3.13.4</atlassian.product.test-lib.version>
 <atlassian.product.data.version>3.13.2</atlassian.product.data.version>
 </properties>

L’altra cosa da fare è modificare i dump di Jira che vorrete usare per i vostri test, in modo da far credere a Jira che sta importando una versione compatibile del dump.
Per fare questo dovete:

1. Aprire il dump xml di Jira che usate per i test (nel nostro caso JiraDataForTest.xml)

2. Cercare l’occorrenza di questa property

<OSPropertyEntry id="12345"
   entityName="jira.properties"
   entityId="1"
   propertyKey="jira.version.patched"
   type="5"/>

Per essere sicuri basta che cerchiate la parola “jira.version.patched”

3. Prendere nota dell’id di questa propery (es 12345) e cercare l’occorrenza di una OSPropertyString con lo stesso id

<OSPropertyString id="12345" value="354"/>

Ecco, quel valore (354) rappresenta la build version di Jira, che per la 3.13.4 è proprio 354.

4. Sostituire il valore 354 con 335, che è la build versione di Jira 3.13.2 e salvare l’xml

5. Rilanciare il test.

Tutto dovrebbe filare liscio ora…

 $ mvn integration-test
 ...
 ...
 [INFO] [jar:jar]
 [INFO] Building jar:
    /Users/pietrodibello/Documents/workspace/MyWonderfulProjectToSaveTheWorld/MyWonderfulPluginToSaveTheWorld/
    target/MyWonderfulPluginToSaveTheWorld-1.0-SNAPSHOT.jar
 [INFO] [antrun:run {execution: generate-integration-test-config}]
 [INFO] Executing tasks
 [propertyfile] Updating property file:
   /Users/pietrodibello/Documents/workspace/MyWonderfulProjectToSaveTheWorld/MyWonderfulPluginToSaveTheWorld/
   target/test-classes/localtest.properties
 [INFO] Executed tasks
 [INFO] [antrun:run {execution: pre-integration-test-user-ant-tasks}]
 [INFO] Executing tasks
 [INFO] Executed tasks
 [INFO] [atlassian-test-harness:start-fisheye {execution: start-fisheye}]
 [INFO] Skipping fisheye; startService is set to false
 [INFO] [atlassian-test-harness:start-confluence {execution: start-confluence}]
 [INFO] Skipping confluence; startService is set to false
 [INFO] [atlassian-test-harness:start-jira {execution: start-jira}]
 [INFO] Output log is set to
   /Users/pietrodibello/Documents/workspace/MyWonderfulProjectToSaveTheWorld/MyWonderfulPluginToSaveTheWorld/target/jira/output.log
 [INFO] Finished with jira goal
 [INFO] [atlassian-test-harness:start-bamboo {execution: start-bamboo}]
 [INFO] Skipping bamboo; startService is set to false
 [INFO] [surefire:test {execution: acceptance_tests}]
 [INFO] Surefire report directory:
   /Users/pietrodibello/Documents/workspace/MyWonderfulProjectToSaveTheWorld/MyWonderfulPluginToSaveTheWorld/target/surefire-reports

 -------------------------------------------------------
 T E S T S
 -------------------------------------------------------
 Running it.com.sourcesense.jira.plugin.JiraTest
 .
 . Started it.com.sourcesense.jira.plugin.JiraTest.test. Wed Jul 08 14:30:44 CEST 2009
 going to page secure/admin/XmlRestore!default.jspa
 Asserting text present: Your project has been successfully imported
 Asserting text present: This JIRA site is for demonstration purposes only
 .
 . Finished it.com.sourcesense.jira.plugin.JiraTest.test. Wed Jul 08 14:30:54 CEST 2009
 . The test ran in 10.542 seconds
 . The test suite has been running for 10.536 seconds
 . Max Mem : 66650112 Total Mem : 2727936 Free Mem : 268968
 . ______________________________
 Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.045 sec

 Results :

 Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

 [INFO] ------------------------------------------------------------------------
 [INFO] BUILD SUCCESSFUL
 [INFO] ------------------------------------------------------------------------
 [INFO] Total time: 44 seconds
 [INFO] Finished at: Wed Jul 08 14:30:55 CEST 2009
 [INFO] Final Memory: 32M/254M
 [INFO] ------------------------------------------------------------------------

Evviva, barra verde!!