System of a test: Proper browser testing in Ruby on Rails — Martian Chronicles


Discover the collection of best practices for end-to-end browser testing of Ruby on Rails applications and adopt them in your projects. See how to ditch Java-based Selenium in favor of leaner, meaner Ferrum-Cuprite combo that uses Chrome DevTools Protocol directly through pure Ruby. And if you use Docker for development—we’ve got you covered too!

Ruby community is passionate about testing. We have plenty of testing libraries, there are hundreds of blog posts on the topic, we even had a dedicated podcast. Hell, the top 3 most downloaded gems are parts of the RSpec testing framework!

Rails, I believe, is one of the reasons behind the rise of Ruby testing. The framework makes writing tests as enjoyable as possible. In most situations, following the comprehensive Rails testing guide is good enough (at least, in the beginning). But there is always an exception, and, in our case, it is system testing.

Writing and maintaining system tests for Rails applications can hardly be called “enjoyable”. My approach to handling this problem evolved a lot since my first Cucumber-driven test suite back in 2013. And today, in 2020, I finally reached the point when I’m ready to share my current setup with everyone. In this post, I’m going to cover the following topics:

System tests in a nutshell

“System tests” is a common naming for automated end-to-end tests in the Rails world. Before Rails adopted this name, we used such variations as feature tests, browser tests, and even acceptance tests (though the latter are ideologically different).

If we recall the testing pyramid (or diamond), system tests live on the very top: they treat the whole program as a black box, usually emulating end-users activities and expectations. And that is why, in case of web applications, we need a browser to run such tests (or at least an emulation like Rack Test).

Let’s take a look at the typical system tests architecture:

System tests architecture

System tests architecture

That’s not the only way of writing end-to-end tests in Rails. For example, you can use Cypress JS framework and IDE. The only reason stopping me from trying this approach is the lack of multiple sessions support, which is required for testing real-time applications (i.e., those with AnyCable 😉).

We need to manage at least three “processes” (some of them could be Ruby threads): a web server running our application, a browser and a test runner itself. That’s the bare minimum. In practice, we usually also need another tool to provide an API to control the browser (e.g., ChromeDriver). There were attempts to simplify this setup by building specific browsers (such as capybara-webkit and PhantomJS) providing such APIs out-of-box, but none of them survived the compatibility race with real browsers.

And, of course, we need to add a handful of Ruby gems to our test dependencies—to glue all pieces together. More dependencies—more problems. For example, Database Cleaner for a long time was a must-have add-on: we couldn’t use transactions to automatically rollback the database state, because each thread used its own connection; we had to use TRUNCATE ... or DELETE FROM ... for each table instead, which is much slower. We solved this problem by using a shared connection in all threads (via the TestProf extension). Rails 5.1 was released with a similar functionality out-of-the-box.

Thus, by adding system tests, we increase the maintenance costs for development and CI environments and introduce potential points of failures or instability: due to the complex setup, flakiness is the most common problem with end-to-end testing. And most of this flakiness comes from communication with a browser.

My standard system_helper.rb as of 2019 contained more than 200 lines of code! It’s still valid if you want to continue using Selenium.

Although by introducing system tests in 5.1 Rails simplified keeping browser tests, they still require some configuration efforts to work smoothly:

  • You need to deal with web drivers (Rails assumes that you use Selenium).
  • You’re on your own with configuring system tests in a containerized environment (i.e., when using Docker for development).
  • The configuration is not flexible enough (e.g., screenshots path).

Let’s move closer to code and see how you can make system testing more fun in 2020!

Modern system tests with Cuprite

By default, Rails assumes that you will be running system tests with Selenium. Selenium is a battle-proven software for web browsers automation. It aims to provide a universal API for all browsers as well as the most realistic experience. Only real humans made of meat and bones could do better in terms of emulating user-browser interactions.

This power doesn’t come for free: you need to install browser-specific drivers, the overhead of realistic interactions is noticeable at scale (i.e., Selenium tests are usually pretty slow).

Selenium was created a while ago, at the times when browsers didn’t provide any built-in capabilities for automation. Several years ago, this situation changed with the introduction of CDP protocol for Chrome. Using CDP, you can manipulate browser sessions directly, no need for intermediate abstraction layers and tools.

A lot of projects leveraging CDP appeared since then, including the most well-known one—Puppeteer, a browser automation library for Node.js. What about the Ruby world? Ferrum, a CDP library for Ruby, although being a pretty young one, provides a comparable to Puppeteer experience. And, what’s more important for us, it ships with a companion project called Cuprite—a pure Ruby Capybara driver using CDP.

I’ve started using Cuprite actively only in the beginning of 2020 (I tried it a year before but had some problems with Docker environment) and I never regret it. Setting up system tests became bloody simple (all you need is loveChrome), and the execution is so fast that after migrating from Selenium some of my tests failed: they lacked proper async expectations and passed in Selenium only because it was much slower.

Let’s take a look at the most recent system tests configuration with Cuprite I worked on.

Annotated configuration example

This example borrows from my recent open-source Ruby on Rails project-AnyCable Rails Demo. It aims to demonstrate how to use the just released AnyCable 1.0 with Rails apps, but we can also use it for this post—it has a decent system tests coverage.

The project uses RSpec and its system testing wrapper. Most of the ideas could be applied to Minitest as well.

Let’s start with a minimal example enough to run tests on a local machine. This code lives in the demo/dockerless branch of AnyCable Rails Demo.

Let’s take a quick at the Gemfile first:

group :test do gem 'capybara' gem 'selenium-webdriver' gem 'cuprite'
end

What? Why do we need selenium-webdriver if we’re not using Selenium at all? It turns out that Rails requires this gem to be present independently of a driver you use. Hopefully, that has been fixed and we’ll be able to remove this gem in Rails 6.1.

I keep system tests configuration in multiple files in the spec/system/support folder and use a dedicated system_helper.rb to load them:

spec/ system/ support/ better_rails_system_tests.rb capybara_setup.rb cuprite_setup.rb precompile_assets.rb ... system_helper.rb

Let’s take a look at each file from the list above and see what it is for.

system_helper.rb

The system_helper.rb file may contain some general RSpec configuration for system tests, but, usually, it’s as simple as the following:

# Load general RSpec Rails configuration
require "rails_helper.rb" # Load configuration files and helpers
Dir[File.join(__dir__, "system/support/**/*.rb")].sort.each { |file| require file }

Then, in your system specs, you use require "system_helper" to activate this configuration.

We use a separate helper file and a support folder for system tests to avoid all the excess configuration in the case when we only need to run a single unit test.

capybara_setup.rb

This file contains configuration for Capybara framework:

# spec/system/support/capybara_setup.rb # Usually, especially when using Selenium, developers tend to increase the max wait time.
# With Cuprite, there is no need for that.
# We use a Capybara default value here explicitly.
Capybara.default_max_wait_time = 2 # Normalize whitespaces when using `has_text?` and similar matchers,
# i.e., ignore newlines, trailing spaces, etc.
# That makes tests less dependent on slightly UI changes.
Capybara.default_normalize_ws = true # Where to store system tests artifacts (e.g. screenshots, downloaded files, etc.).
# It could be useful to be able to configure this path from the outside (e.g., on CI).
Capybara.save_path = ENV.fetch("CAPYBARA_ARTIFACTS", "./tmp/capybara")

This file also contains a useful patch for Capybara, which purpose we will reveal later:

# spec/system/support/capybara_setup.rb Capybara.singleton_class.prepend(Module.new do attr_accessor :last_used_session def using_session(name, &block) self.last_used_session = name super ensure self.last_used_session = nil end
end)

The Capybara.using_session allows you to manipulate a different browser session, and thus, multiple independent sessions within a single test scenario. That’s especially useful for testing real-time features, e.g., something with WebSocket.

This patch tracks the name of the last session used. We’re going to use this information to support taking failure screenshots in multi-session tests.

cuprite_setup.rb

This file is responsible for configuring Cuprite:

# spec/system/support/cuprite_setup.rb # First, load Cuprite Capybara integration
require "capybara/cuprite" # Then, we need to register our driver to be able to use it later
# with #driven_by method.
Capybara.register_driver(:cuprite) do |app| Capybara::Cuprite::Driver.new( app, **{ window_size: [1200, 800], # See additional options for Dockerized environment in the respective section of this article browser_options: {}, # Increase Chrome startup wait time (required for stable CI builds) process_timeout: 10, # Enable debugging capabilities inspector: true, # Allow running Chrome in a headful mode by setting HEADLESS env # var to a falsey value headless: !ENV["HEADLESS"].in?(%w[n 0 no false]) } )
end # Configure Capybara to use :cuprite driver by default
Capybara.default_driver = Capybara.javascript_driver = :cuprite

We also define a few shortcuts for common Cuprite API methods:

module CupriteHelpers # Drop #pause anywhere in a test to stop the execution. # Useful when you want to checkout the contents of a web page in the middle of a test # running in a headful mode. def pause page.driver.pause end # Drop #debug anywhere in a test to open a Chrome inspector and pause the execution def debug(*args) page.driver.debug(*args) end
end RSpec.configure do |config| config.include CupriteHelpers, type: :system
end

Below you can see a demonstration of how this #debug helper works.

Debugging system tests

better_rails_system_tests.rb

This file contains some patches to Rails system tests internals as well as some general configuration (see code comments for explanations):

# spec/system/support/better_rails_system_tests.rb module BetterRailsSystemTests # Use our `Capybara.save_path` to store screenshots with other capybara artifacts # (Rails screenshots path is not configurable https://github.com/rails/rails/blob/49baf092439fc74fc3377b12e3334c3dd9d0752f/actionpack/lib/action_dispatch/system_testing/test_helpers/screenshot_helper.rb#L79) def absolute_image_path Rails.root.join("#{Capybara.save_path}/screenshots/#{image_name}.png") end # Make failure screenshots compatible with multi-session setup. # That's where we use Capybara.last_used_session introduced before. def take_screenshot return super unless Capybara.last_used_session Capybara.using_session(Capybara.last_used_session) { super } end
end RSpec.configure do |config| config.include BetterRailsSystemTests, type: :system # Make urls in mailers contain the correct server host. # It's required for testing links in emails (e.g., via capybara-email). config.around(:each, type: :system) do |ex| was_host = Rails.application.default_url_options[:host] Rails.application.default_url_options[:host] = Capybara.server_host ex.run Rails.application.default_url_options[:host] = was_host end # Make sure this hook runs before others config.prepend_before(:each, type: :system) do # Use JS driver always driven_by Capybara.javascript_driver end
end

precompile_assets.rb

This file is responsible for precompiling assets before running system tests (I’m not going to paste it here in full, just the most interesting parts):

RSpec.configure do |config| # Skip assets precompilcation if we exclude system specs. # For example, you can run all non-system tests via the following command: # # rspec --tag ~type:system # # In this case, we don't need to precompile assets. next if config.filter.opposite.rules[:type] == "system" || config.exclude_pattern.match?(%r{spec/system}) config.before(:suite) do # We can use webpack-dev-server for tests, too! # Useful if you working on a frontend code fixes and want to verify them via system tests. if Webpacker.dev_server.running? $stdout.puts "\n⚙️ Webpack dev server is running! Skip assets compilation.\n" next else $stdout.puts "\n🐢 Precompiling assets.\n" # The code to run webpacker:compile Rake task # ... end end
end

Why precompiling assets manually if Rails can do that for you automatically? The problem is that Rails precompiles assets lazily (i.e., the first time you request an asset), that could make the first test example running much slower and even encounter random timeout exceptions.

To use Webpack dev server for tests, you need to add a dev_server configuration for test environment to your webpacker.yml and run it via RAILS_ENV=test ./bin/webpack-dev-server.

Another thing I want to pay attention to is the ability to use a Webpack dev server for system tests. That’s really useful when you’re working hard on frontend code refactoring: you can pause a test, open a browser, edit frontend code and see it hot reloaded!

Dockerizing system tests

Let’s move our configuration to the next level and make it compatible with our Docker development environment. The dockerized version of the test setup above lives in the default branch of the AnyCable Rails Demo repository, feel free to check it out, but we’re going to cover all the interesting bits below.

The main difference in a Docker setup is that we run a browser instance in a separate container. It’s possible to add Chrome to your base Rails image or, probably, even use a host machine browser from a container (like it could be done with Selenium and ChromeDriver). But, in my opinion, defining a dedicated browser service to docker-compose.yml is a proper Docker way of doing this.

Currently, I’m using a Chrome Docker image from browserless.io. It comes with a handy Debug Viewer, which allows you to debug headless browser sessions (wait for short video at the end of the article):

services: # ... chrome: image: browserless/chrome:1.31-chrome-stable ports: - "3333:3333" environment: # By default, it uses 3000, which is typically used by Rails. PORT: 3333 # Set connection timeout to avoid timeout exception during debugging # https://docs.browserless.io/docs/docker.html#connection-timeout CONNECTION_TIMEOUT: 600000

Add CHROME_URL: http://chrome:3333 to your Rails service environment and run Chrome in the background:

docker-compose up -d chrome

Now we need to configure Cuprite to work with a remote browser if its URL is provided:

# cuprite_setup.rb # Parse URL
# NOTE: REMOTE_CHROME_HOST should be added to Webmock/VCR allowlist if you use any of those.
REMOTE_CHROME_URL = ENV["CHROME_URL"]
REMOTE_CHROME_HOST, REMOTE_CHROME_PORT = if REMOTE_CHROME_URL URI.parse(REMOTE_CHROME_URL).yield_self do |uri| [uri.host, uri.port] end end # Check whether the remote chrome is running.
remote_chrome = begin if REMOTE_CHROME_URL.nil? false else Socket.tcp(REMOTE_CHROME_HOST, REMOTE_CHROME_PORT, connect_timeout: 1).close true end rescue Errno::ECONNREFUSED, Errno::EHOSTUNREACH, SocketError false end remote_options = remote_chrome ? { url: REMOTE_CHROME_URL } : {}

This configuration above assumes that a user wants to use a locally installed Chrome if CHROME_URL is not set or browser is not responding.

We do that to make our configuration backward-compatible with the local one (we usually do not force everyone to use Docker for development; we let Docker deniers suffer with their unique local setups 😈).

Our driver registration now looks like this:

# spec/system/support/cuprite_setup.rb Capybara.register_driver(:cuprite) do |app| Capybara::Cuprite::Driver.new( app, **{ window_size: [1200, 800], browser_options: remote_chrome ? { "no-sandbox" => nil } : {}, inspector: true }.merge(remote_options) )
end

We also need to update our #debug helper to print a Debug Viewer URL instead of trying to open a browser:

module CupriteHelpers # ... def debug(binding = nil) $stdout.puts "🔎 Open Chrome inspector at http://localhost:3333" return binding.pry if binding page.driver.pause end
end

Since a browser is running on a different “machine”, it should know how to reach the test server (which no longer listens on localhost).

For that, we need to configure Capybara server host:

# spec/system/support/capybara_setup.rb # Make server accessible from the outside world
Capybara.server_host = "0.0.0.0"
# Use a hostname that could be resolved in the internal Docker network
# NOTE: Rails overrides Capybara.app_host in Rails <6.1, so we have
# to store it differently
CAPYBARA_APP_HOST = `hostname`.strip&.downcase || "0.0.0.0"
# In Rails 6.1+ the following line should be enough
# Capybara.app_host = "http://#{`hostname`.strip&.downcase || "0.0.0.0"}"

Finally, let’s add a couple of tweaks to our better_rails_system_tests.rb.

First, let’s make screenshot notices clickable in VS Code 🙂 (Docker absolute paths are different from a host system):

# spec/system/support/better_rails_system_tests.rb module BetterRailsSystemTests # ... # Use relative path in screenshot message def image_path absolute_image_path.relative_path_from(Rails.root).to_s end
end

Secondly, make sure tests use the correct server host (this has been fixed in Rails 6.1):

# spec/system/support/better_rails_system_tests.rb config.prepend_before(:each, type: :system) do # Rails sets host to `127.0.0.1` for every test by default. # That won't work with a remote browser. host! CAPYBARA_APP_HOST # Use JS driver always driven_by Capybara.javascript_driver
end

In too Dip

If you use Dip to manage your dockerized development environment (and I highly recommend you do, it gives you yield the power of containers without the overhead of remembering all Docker CLI Commands), you can avoid launching chrome service manually by adding custom commands to your dip.yml and an additional service definition to your docker-compose.yml:

# docker-compose.yml # Separate definition for system tests to add Chrome as a dependency
rspec_system: <<: *backend depends_on: <<: *backend_depends_on chrome: condition: service_started # dip.yml
rspec: description: Run Rails unit tests service: rails environment: RAILS_ENV: test command: bundle exec rspec --exclude-pattern spec/system/**/*_spec.rb subcommands: system: description: Run Rails system tests service: rspec_system command: bundle exec rspec --pattern spec/system/**/*_spec.rb

Now, to run system tests I use the following command:

And that’s it!

As a final note, let me show you how debugging with the Debug Viewer from Browserless.io Docker image looks like:

Debugging system tests running in Docker

If you want to set up rock-solid development practices in your engineering team—regardless of the technical stack—feel free to drop us a line. Helping companies to improve their engineering culture is one of the favorite things we do!